[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 496702076 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002397] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.006805] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009173] LSM: Security Framework initializing [ 0.010089] Yama: becoming mindful. [ 0.011069] SELinux: Initializing. [ 0.012081] *** VALIDATE selinux *** [ 0.020681] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025039] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026181] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027144] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028158] *** VALIDATE tmpfs *** [ 0.030138] *** VALIDATE proc *** [ 0.031281] *** VALIDATE cgroup *** [ 0.032012] *** VALIDATE cgroup2 *** [ 0.034155] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035169] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038018] Spectre V2 : User space: Vulnerable [ 0.039011] Speculative Store Bypass: Vulnerable [ 0.042839] debug: unmapping init [mem 0xffffffff92259000-0xffffffff92260fff] [ 0.045190] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046690] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047037] ... version: 2 [ 0.047887] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049016] ... value mask: 0000ffffffffffff [ 0.050016] ... max period: 00007fffffffffff [ 0.051017] ... fixed-purpose events: 3 [ 0.052017] ... event mask: 000000070000000f [ 0.053401] rcu: Hierarchical SRCU implementation. [ 0.055606] smp: Bringing up secondary CPUs ... [ 0.056541] x86: Booting SMP configuration: [ 0.057036] .... node #0, CPUs: #1 #2 #3 [ 0.061013] smp: Brought up 1 node, 4 CPUs [ 0.062923] smpboot: Max logical packages: 1 [ 0.063027] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.141568] node 0 deferred pages initialised in 76ms [ 0.145219] devtmpfs: initialized [ 0.147348] x86/mm: Memory block size: 128MB [ 0.150990] gcov: version magic: 0x41383552 [ 0.154366] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.155081] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.158386] pinctrl core: initialized pinctrl subsystem [ 0.160203] [ 0.160989] ************************************************************* [ 0.163013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.165015] ** ** [ 0.168017] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.171016] ** ** [ 0.173015] ** This means that this kernel is built to expose internal ** [ 0.176018] ** IOMMU data structures, which may compromise security on ** [ 0.178014] ** your system. ** [ 0.181018] ** ** [ 0.185019] ** If you see this message and you are not debugging the ** [ 0.187014] ** kernel, report this immediately to your vendor! ** [ 0.189016] ** ** [ 0.192018] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.194020] ************************************************************* [ 0.197416] NET: Registered protocol family 16 [ 0.199435] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.202073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.209701] cpuidle: using governor menu [ 0.212403] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.215323] PCI: Using configuration type 1 for base access [ 0.217136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.227052] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229047] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233151] cryptd: max_cpu_qlen set to 1000 [ 0.237245] ACPI: Added _OSI(Module Device) [ 0.239015] ACPI: Added _OSI(Processor Device) [ 0.241013] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.243024] ACPI: Added _OSI(Processor Aggregator Device) [ 0.248928] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.256420] ACPI: Interpreter enabled [ 0.258060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260014] ACPI: Using IOAPIC for interrupt routing [ 0.262101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.266379] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.277944] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.280062] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.283026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.287110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.293566] acpiphp: Slot [2] registered [ 0.295220] acpiphp: Slot [5] registered [ 0.296200] acpiphp: Slot [6] registered [ 0.298159] acpiphp: Slot [7] registered [ 0.300154] acpiphp: Slot [8] registered [ 0.301198] acpiphp: Slot [9] registered [ 0.303218] acpiphp: Slot [10] registered [ 0.305114] acpiphp: Slot [3] registered [ 0.307122] acpiphp: Slot [4] registered [ 0.309105] acpiphp: Slot [11] registered [ 0.311095] acpiphp: Slot [12] registered [ 0.312106] acpiphp: Slot [13] registered [ 0.314113] acpiphp: Slot [14] registered [ 0.316131] acpiphp: Slot [15] registered [ 0.317138] acpiphp: Slot [16] registered [ 0.319108] acpiphp: Slot [17] registered [ 0.321108] acpiphp: Slot [18] registered [ 0.322118] acpiphp: Slot [19] registered [ 0.324113] acpiphp: Slot [20] registered [ 0.326135] acpiphp: Slot [21] registered [ 0.328128] acpiphp: Slot [22] registered [ 0.329136] acpiphp: Slot [23] registered [ 0.331237] acpiphp: Slot [24] registered [ 0.333143] acpiphp: Slot [25] registered [ 0.334155] acpiphp: Slot [26] registered [ 0.336149] acpiphp: Slot [27] registered [ 0.338141] acpiphp: Slot [28] registered [ 0.340153] acpiphp: Slot [29] registered [ 0.342150] acpiphp: Slot [30] registered [ 0.344157] acpiphp: Slot [31] registered [ 0.346081] PCI host bridge to bus 0000:00 [ 0.347029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.350036] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.352030] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.355034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.358040] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.361048] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.363223] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.367075] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.371422] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.381797] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.386770] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.389019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.392023] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.394018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.397703] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.400897] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.404060] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.407964] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.414025] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.426020] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.433019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.440393] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.448015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.455022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.470024] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.480487] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.488023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.494260] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.512020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.525108] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.531020] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.537019] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.550022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.560504] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.570022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.576020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.597019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.610833] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.618022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.623021] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.642000] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.653854] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.663020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.669023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.688022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.698000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.700343] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.702336] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.704363] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.707375] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.713081] iommu: Default domain type: Passthrough [ 0.714693] SCSI subsystem initialized [ 0.716255] ACPI: bus type USB registered [ 0.718236] usbcore: registered new interface driver usbfs [ 0.721140] usbcore: registered new interface driver hub [ 0.723115] usbcore: registered new device driver usb [ 0.725333] pps_core: LinuxPPS API ver. 1 registered [ 0.727015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.731093] PTP clock support registered [ 0.733382] EDAC MC: Ver: 3.0.0 [ 0.736233] PCI: Using ACPI for IRQ routing [ 0.739251] NetLabel: Initializing [ 0.740016] NetLabel: domain hash size = 128 [ 0.742017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.744116] NetLabel: unlabeled traffic allowed by default [ 0.747360] vgaarb: loaded [ 0.748358] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.751018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.759000] clocksource: Switched to clocksource kvm-clock [ 0.870825] VFS: Disk quotas dquot_6.6.0 [ 0.872395] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.875074] *** VALIDATE ramfs *** [ 0.876498] *** VALIDATE hugetlbfs *** [ 0.878307] pnp: PnP ACPI init [ 0.881496] pnp: PnP ACPI: found 6 devices [ 0.896327] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.899437] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.901585] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.903698] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.906463] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.909538] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.913083] NET: Registered protocol family 2 [ 0.916421] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.921886] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.926425] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.932526] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.937109] TCP: Hash tables configured (established 65536 bind 65536) [ 0.940822] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.944699] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.948441] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.952370] NET: Registered protocol family 1 [ 0.957050] RPC: Registered named UNIX socket transport module. [ 0.959612] RPC: Registered udp transport module. [ 0.961538] RPC: Registered tcp transport module. [ 0.963714] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.967051] NET: Registered protocol family 44 [ 0.969302] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.972153] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.974584] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.976856] PCI: CLS 0 bytes, default 64 [ 0.978811] Unpacking initramfs... [ 2.382675] debug: unmapping init [mem 0xffff88c9bcc54000-0xffff88c9bffbffff] [ 2.389391] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.391587] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.395281] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.934185] Initialise system trusted keyrings [ 2.936158] Key type blacklist registered [ 2.939083] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.948271] zbud: loaded [ 2.951935] *** VALIDATE nfs *** [ 2.953364] *** VALIDATE nfs4 *** [ 2.955869] pstore: using deflate compression [ 2.963120] Platform Keyring initialized [ 3.074929] NET: Registered protocol family 38 [ 3.077271] Key type asymmetric registered [ 3.079151] Asymmetric key parser 'x509' registered [ 3.081276] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.085277] io scheduler mq-deadline registered [ 3.087137] io scheduler kyber registered [ 3.089175] io scheduler bfq registered [ 3.091231] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.094358] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.096873] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.100485] ACPI: Power Button [PWRF] [ 3.106364] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.113080] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.125547] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.132433] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.148693] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.179529] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.209035] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.214161] Non-volatile memory driver v1.3 [ 3.216247] Linux agpgart interface v0.103 [ 3.246568] virtio_blk virtio1: [vda] 146144 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.250134] vda: detected capacity change from 0 to 74825728 [ 3.269098] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.272549] vdb: detected capacity change from 0 to 1073741824 [ 3.287943] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.291393] vdc: detected capacity change from 0 to 2621440000 [ 3.309163] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.312491] vdd: detected capacity change from 0 to 2621440000 [ 3.340567] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.344448] vde: detected capacity change from 0 to 4294967296 [ 3.364989] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.368194] vdf: detected capacity change from 0 to 4294967296 [ 3.373857] libphy: Fixed MDIO Bus: probed [ 3.387542] usbcore: registered new interface driver usbserial_generic [ 3.389618] usbserial: USB Serial support registered for generic [ 3.391552] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.396478] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.398492] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.401242] mousedev: PS/2 mouse device common for all mice [ 3.404424] rtc_cmos 00:05: RTC can wake from S4 [ 3.408805] rtc_cmos 00:05: registered as rtc0 [ 3.411346] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.411421] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.419079] intel_pstate: CPU model not supported [ 3.422177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.429577] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.431428] hid: raw HID events driver (C) Jiri Kosina [ 3.440088] usbcore: registered new interface driver usbhid [ 3.442714] usbhid: USB HID core driver [ 3.444801] drop_monitor: Initializing network drop monitor service [ 3.448248] Initializing XFRM netlink socket [ 3.450667] NET: Registered protocol family 10 [ 3.454577] Segment Routing with IPv6 [ 3.456366] NET: Registered protocol family 17 [ 3.458424] mpls_gso: MPLS GSO support [ 3.463229] RAS: Correctable Errors collector initialized. [ 3.466025] AVX version of gcm_enc/dec engaged. [ 3.468230] AES CTR mode by8 optimization enabled [ 3.561512] sched_clock: Marking stable (3561487708, 0)->(4480288384, -918800676) [ 3.566886] registered taskstats version 1 [ 3.570749] Loading compiled-in X.509 certificates [ 3.574214] zswap: loaded using pool lzo/zbud [ 3.597655] Key type big_key registered [ 3.609881] Key type encrypted registered [ 3.611633] ima: No TPM chip found, activating TPM-bypass! [ 3.614043] ima: Allocated hash algorithm: sha1 [ 3.615930] ima: No architecture policies found [ 3.617787] evm: Initialising EVM extended attributes: [ 3.619911] evm: security.selinux [ 3.621260] evm: security.ima [ 3.622495] evm: security.capability [ 3.623816] evm: HMAC attrs: 0x1 [ 3.626369] rtc_cmos 00:05: setting system clock to 2026-08-22 21:04:53 UTC (1787432693) [ 3.632225] debug: unmapping init [mem 0xffffffff93203000-0xffffffff933fffff] [ 3.635422] debug: unmapping init [mem 0xffffffff91f82000-0xffffffff92258fff] [ 3.645067] Write protecting the kernel read-only data: 28672k [ 3.648490] debug: unmapping init [mem 0xffffffff90603000-0xffffffff907fffff] [ 3.652238] debug: unmapping init [mem 0xffffffff90f14000-0xffffffff90ffffff] [ 3.687915] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.700505] systemd[1]: Detected virtualization kvm. [ 3.702712] systemd[1]: Detected architecture x86-64. [ 3.704742] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.733853] systemd[1]: No hostname configured. [ 3.735436] systemd[1]: Set hostname to . [ 3.737221] random: systemd: uninitialized urandom read (16 bytes read) [ 3.739368] systemd[1]: Initializing machine ID from random generator. [ 3.777396] random: ln: uninitialized urandom read (6 bytes read) [ 3.868097] random: systemd: uninitialized urandom read (16 bytes read) [ 3.871661] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.876699] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.881735] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. Starting Journal Service... Starting Create Volatile Files and Directories... [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.465346] device-mapper: uevent: version 1.0.3 [ 4.467806] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.140477] virtio_net virtio0 ens2: renamed from eth0 [ 5.144386] random: fast init done [ 5.198029] scsi host0: ata_piix [ 5.211589] scsi host1: ata_piix [ 5.213670] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.217396] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.750723] dracut-initqueue[578]: RTNETLINK answers: File exists [ 10.133666] random: crng init done [ 10.135378] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.570070] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.653332] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.892245] SELinux: Disabled at runtime. [ 11.950990] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.960534] systemd[1]: Detected virtualization kvm. [ 11.962641] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.449820] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.452756] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.457544] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.463281] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.467281] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.474305] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.482247] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ 12.520918] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.916411] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.202076] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.243080] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.331530] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.345732] EDAC sbridge: Ver: 1.1.2 [ 15.176985] Key type dns_resolver registered [ 15.730784] NFS: Registering the id_resolver key type [ 15.735294] Key type id_resolver registered [ 15.738169] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg315-server login: [ 63.890075] libcfs: loading out-of-tree module taints kernel. [ 63.940483] Key type ._llcrypt registered [ 63.942691] Key type .llcrypt registered [ 64.032822] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 80.733701] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 82.445349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 82.462836] alg: No test for adler32 (adler32-zlib) [ 84.071251] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 85.048943] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 86.855375] Key type lgssc registered [ 88.618699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 96.763109] hrtimer: interrupt took 2192821 ns [ 105.793484] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 144.670402] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 158.342613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.391567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.838730] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 159.890252] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 160.028211] Lustre: lustre-MDT0000: new disk, initializing [ 160.162702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.198855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.595413] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.678069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.750765] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.792253] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.799179] Lustre: Skipped 1 previous similar message [ 177.891879] Lustre: lustre-MDT0001: new disk, initializing [ 177.973489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 178.054792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 178.085243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.534689] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.147979] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 195.922595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.201299] Lustre: lustre-OST0000: new disk, initializing [ 196.204775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 196.211989] Lustre: 8439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.264061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 202.273234] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 204.310903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 204.321982] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 204.395968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 216.436881] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 216.636782] Lustre: lustre-OST0001: new disk, initializing [ 216.639895] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 216.652743] Lustre: 9511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 216.776449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.689702] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.243955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.255703] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.303189] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 234.600152] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 238.717900] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 244.761876] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing check_logdir /tmp/testlogs/ [ 249.311671] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing yml_node [ 253.678943] Lustre: DEBUG MARKER: Client: 2.17.57.64 [ 256.088993] Lustre: DEBUG MARKER: MDS: 2.17.57.64 [ 258.461177] Lustre: DEBUG MARKER: OSS: 2.17.57.64 [ 260.071949] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Aug 22 17:09:08 EDT 2026 [ 275.038736] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 276.402601] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 294.880463] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 294.892182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 294.893222] Lustre: Skipped 3 previous similar messages [ 294.921601] Lustre: Skipped 3 previous similar messages [ 300.005037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.120491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.126266] Lustre: Skipped 6 previous similar messages [ 307.167121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 307.405703] Lustre: server umount lustre-MDT0000 complete [ 310.604778] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433000 with bad export cookie 13027591689575318368 [ 310.611103] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 310.611865] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 315.360823] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.361986] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.366049] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.386210] LustreError: 6508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 315.418313] Lustre: Skipped 1 previous similar message [ 317.062221] Lustre: server umount lustre-MDT0001 complete [ 321.018813] Lustre: server umount lustre-OST0000 complete [ 324.892618] Lustre: server umount lustre-OST0001 complete [ 331.565231] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 340.834300] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 382.586985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 382.764764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 382.795812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 382.858305] Lustre: lustre-MDT0000: new disk, initializing [ 382.944393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 382.962057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 387.376800] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 398.325900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 398.453304] Lustre: 15066:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 398.523705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 398.527576] Lustre: Skipped 1 previous similar message [ 398.613635] Lustre: lustre-MDT0001: new disk, initializing [ 398.701667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 398.746402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 398.755791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 402.694811] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 412.144503] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 418.435172] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 424.313871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 424.597676] Lustre: lustre-OST0000: new disk, initializing [ 424.603992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 424.611916] Lustre: 17037:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 424.732403] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 426.450866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 426.463476] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 426.623138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 430.030919] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 439.715953] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 444.707102] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 444.975376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 449.068559] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 449.340311] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 452.068265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 452.079139] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 452.080864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 452.097571] Lustre: Skipped 1 previous similar message [ 452.104610] Lustre: Skipped 1 previous similar message [ 462.308854] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 462.312421] Lustre: Skipped 3 previous similar messages [ 465.375397] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 465.586495] Lustre: server umount lustre-OST0000 complete [ 470.496575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 470.503072] LustreError: Skipped 1 previous similar message [ 470.508373] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 475.556979] Lustre: server umount lustre-MDT0000 complete [ 477.677395] LustreError: 15876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.691690] LustreError: 15876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 479.334236] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433169 with bad export cookie 13027591689575319908 [ 479.338395] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 479.355144] LustreError: 15058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 479.965938] Lustre: server umount lustre-MDT0001 complete [ 484.338485] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:12:52 (1787433172) [ 487.082492] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 488.933802] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:12:56 (1787433176) [ 495.558803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 495.846426] LustreError: 19448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 495.885857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 499.412790] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 501.219780] LustreError: 19449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 505.313564] LustreError: 19448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.851476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.321832] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 519.752666] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 526.481165] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 534.256735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.960080] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 548.255369] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 559.591617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 559.592846] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 559.596393] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 559.596401] Lustre: Skipped 3 previous similar messages [ 559.637887] Lustre: Skipped 2 previous similar messages [ 560.550370] Lustre: server umount lustre-OST0000 complete [ 564.716985] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 570.144797] Lustre: server umount lustre-MDT0000 complete [ 573.660631] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433263 with bad export cookie 13027591689575320790 [ 573.665099] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 573.679472] LustreError: 19430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 574.090464] Lustre: server umount lustre-MDT0001 complete [ 586.033714] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 588.828273] Key type lgssc unregistered [ 589.117682] LNet: 22983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 589.122475] LNetError: 22983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 589.139062] LNet: Removed LNI 192.168.203.115@tcp [ 589.954088] Key type .llcrypt unregistered [ 589.960542] Key type ._llcrypt unregistered [ 611.169519] Key type ._llcrypt registered [ 611.177583] Key type .llcrypt registered [ 611.292888] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 612.595084] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 612.613364] alg: No test for adler32 (adler32-zlib) [ 613.690726] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 613.962845] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 615.639217] Key type lgssc registered [ 616.841309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 625.408570] loop: module loaded [ 625.469690] loop0: detected capacity change from 0 to 204800000 [ 625.687190] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 625.718598] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 626.062378] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 632.914398] loop0: detected capacity change from 0 to 204800000 [ 633.187968] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.199606] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 633.210461] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 638.447869] loop0: detected capacity change from 0 to 204800000 [ 644.775473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 644.791518] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 646.009347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 646.074708] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 646.180154] Lustre: lustre-MDT0000: new disk, initializing [ 646.320649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 646.374479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 651.521881] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 660.057380] loop1: detected capacity change from 0 to 204800000 [ 665.385282] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 665.556864] Lustre: lustre-OST07c6: new disk, initializing [ 665.559750] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 665.562994] Lustre: Skipped 1 previous similar message [ 665.567723] Lustre: 26169:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 665.632628] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 670.813832] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 671.272905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 671.280401] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 671.301572] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 676.048628] Lustre: 25154:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 676.063921] Lustre: 25154:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 676.068904] Lustre: 25154:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 676.075465] Lustre: 25154:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 676.081074] Lustre: 25154:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 676.088603] Lustre: 25154:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 693.457207] Lustre: 26177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 693.462382] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 693.467453] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 693.472352] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 693.479952] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 693.486193] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.242412] Lustre: 26177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 698.252934] Lustre: 26177:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 698.261140] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 698.265142] Lustre: 26177:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.268629] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 698.272834] Lustre: 26177:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.278094] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 698.281737] Lustre: 26177:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.286844] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 698.293192] Lustre: 26177:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 698.302702] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 698.309503] Lustre: 26177:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.460822] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 701.467997] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 701.473369] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 701.478844] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.486378] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 701.490895] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.494772] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 701.507261] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.514707] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 701.520941] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 701.529152] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 701.537102] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 705.996455] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 706.009453] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 706.016276] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 706.023965] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.028411] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 706.035321] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.040554] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 706.045643] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.049563] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 706.054726] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 706.059407] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 706.064552] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 716.685822] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 716.700408] Lustre: 25155:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 716.705744] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 716.711082] Lustre: 25155:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.718282] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 716.724718] Lustre: 25155:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.728897] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 716.735501] Lustre: 25155:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.743894] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 716.751320] Lustre: 25155:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 716.762186] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 716.774099] Lustre: 25155:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 765.640837] Lustre: server umount lustre-OST07c6 complete [ 783.985804] Lustre: server umount lustre-MDT0000 complete [ 799.819673] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 802.458973] Key type lgssc unregistered [ 802.772774] LNet: 28753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 802.792958] LNetError: 28753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 802.812559] LNet: Removed LNI 192.168.203.115@tcp [ 803.762720] Key type .llcrypt unregistered [ 803.766550] Key type ._llcrypt unregistered [ 820.788537] Key type ._llcrypt registered [ 820.791463] Key type .llcrypt registered [ 820.875731] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 821.593034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 821.760261] alg: No test for adler32 (adler32-zlib) [ 822.903591] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 823.190100] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 824.911246] Key type lgssc registered [ 826.102806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 833.336284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 833.366435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.745412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 838.946413] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 844.259086] LustreError: 30018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 844.267992] LustreError: 30018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 846.886932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 847.207280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 850.933496] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 858.426095] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 863.519615] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 869.235588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 869.730000] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.068841] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 880.000757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 885.087415] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 900.581099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.590475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 900.590647] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.602341] Lustre: Skipped 1 previous similar message [ 905.710973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 905.726815] Lustre: Skipped 2 previous similar messages [ 906.298213] Lustre: server umount lustre-OST0000 complete [ 910.820526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.855691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.871121] Lustre: Skipped 1 previous similar message [ 915.798717] Lustre: server umount lustre-MDT0000 complete [ 915.936718] LustreError: 30018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.952216] LustreError: 30018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 918.809352] LustreError: 29998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433608 with bad export cookie 18065457733032517176 [ 918.820824] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.830468] LustreError: 29998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 919.141204] Lustre: server umount lustre-MDT0001 complete [ 928.906736] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 931.465993] Key type lgssc unregistered [ 931.740836] LNet: 33756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.749188] LNetError: 33756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.768367] LNet: Removed LNI 192.168.203.115@tcp [ 932.565887] Key type .llcrypt unregistered [ 932.571718] Key type ._llcrypt unregistered [ 947.732429] Key type ._llcrypt registered [ 947.735206] Key type .llcrypt registered [ 947.807672] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 948.397560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.422376] alg: No test for adler32 (adler32-zlib) [ 949.452134] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 949.645212] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 951.295285] Key type lgssc registered [ 952.329965] Lustre: Echo OBD driver; http://www.lustre.org/ [ 959.120423] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 959.161790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 960.443606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 964.496150] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 970.722615] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 970.776797] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 973.466440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 974.085286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 977.926664] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 984.085825] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 988.918808] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 994.423663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 994.747758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1000.117410] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1005.993618] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1007.175717] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.296822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1016.299591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.301653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1016.327631] Lustre: Skipped 1 previous similar message [ 1021.415967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1021.423217] Lustre: Skipped 2 previous similar messages [ 1022.354298] Lustre: server umount lustre-OST0000 complete [ 1026.534258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1026.547585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1026.556745] Lustre: Skipped 1 previous similar message [ 1031.649360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1031.663253] Lustre: Skipped 1 previous similar message [ 1032.065361] Lustre: server umount lustre-MDT0000 complete [ 1035.408222] LustreError: 34985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433725 with bad export cookie 2106662657371701633 [ 1035.417105] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1035.424774] LustreError: 34985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1035.715064] Lustre: server umount lustre-MDT0001 complete [ 1044.152445] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1046.428629] Key type lgssc unregistered [ 1046.850209] LNet: 38543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1046.859180] LNetError: 38543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1046.874147] LNet: Removed LNI 192.168.203.115@tcp [ 1047.589805] Key type .llcrypt unregistered [ 1047.592119] Key type ._llcrypt unregistered [ 1075.703237] Key type ._llcrypt registered [ 1075.707208] Key type .llcrypt registered [ 1075.826625] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1076.738407] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1076.802839] alg: No test for adler32 (adler32-zlib) [ 1077.858729] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 1078.110974] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1079.799194] Key type lgssc registered [ 1081.045922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1091.352414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1091.419444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1093.099500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.160189] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1102.321140] LustreError: 39962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1102.338125] LustreError: 39962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1107.428988] LustreError: 39961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.783594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1109.186870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.975736] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1120.064745] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1126.936724] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1133.707887] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1134.046497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1138.175186] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1140.214713] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1147.391424] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1155.439155] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 1155.556804] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1155.562857] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1155.759268] Lustre: server umount lustre-MDT0000 complete [ 1158.630213] LustreError: 39962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1158.634440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.686030] Lustre: Skipped 1 previous similar message [ 1159.486947] LustreError: 39942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787433849 with bad export cookie 2520771962683110157 [ 1159.498401] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1159.502465] LustreError: 39942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1160.567576] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 1160.571053] Lustre: Skipped 1 previous similar message [ 1163.745308] LustreError: 39958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.745829] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1163.780867] LustreError: 39958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1163.803473] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1165.886600] Lustre: server umount lustre-MDT0001 complete [ 1187.701385] Lustre: server umount lustre-OST0000 complete [ 1203.946421] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1206.771931] Key type lgssc unregistered [ 1207.079629] LNet: 43827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1207.097069] LNetError: 43827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1207.118515] LNet: Removed LNI 192.168.203.115@tcp [ 1208.015654] Key type .llcrypt unregistered [ 1208.018265] Key type ._llcrypt unregistered [ 1230.428767] Key type ._llcrypt registered [ 1230.431968] Key type .llcrypt registered [ 1230.510484] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1231.214534] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1231.238133] alg: No test for adler32 (adler32-zlib) [ 1232.245422] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 1232.448470] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1234.143847] Key type lgssc registered [ 1234.978680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1241.405256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1241.431490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1242.919216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1247.004385] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1252.323411] LustreError: 45222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1252.346518] LustreError: 45222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1255.511421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1256.147223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1261.529890] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1269.435296] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1274.968662] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1281.473456] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1281.896142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1287.372863] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1288.051298] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1293.260385] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1303.520542] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1303.527069] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1303.540889] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.576856] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1305.576945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1305.588968] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1308.845864] Lustre: server umount lustre-OST0000 complete [ 1326.051675] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.060555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1326.071222] Lustre: Skipped 2 previous similar messages [ 1331.664134] Lustre: server umount lustre-MDT0000 complete [ 1332.192577] LustreError: 45950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1332.208625] LustreError: 45950:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1334.209948] LustreError: 45203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434024 with bad export cookie 10036070552723427929 [ 1334.221368] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1334.229854] LustreError: 45203:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1340.831501] Lustre: server umount lustre-MDT0001 complete [ 1352.915171] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1356.127916] Key type lgssc unregistered [ 1356.492752] LNet: 48933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1356.505854] LNetError: 48933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1356.524240] LNet: Removed LNI 192.168.203.115@tcp [ 1357.393512] Key type .llcrypt unregistered [ 1357.395687] Key type ._llcrypt unregistered [ 1381.856833] Key type ._llcrypt registered [ 1381.859488] Key type .llcrypt registered [ 1381.978884] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1382.995094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1383.036147] alg: No test for adler32 (adler32-zlib) [ 1384.009750] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 1384.317229] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1386.068884] Key type lgssc registered [ 1387.085493] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1393.855733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1393.890775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1395.245196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1399.091863] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1405.414154] LustreError: 50349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1405.436686] LustreError: 50349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1407.082779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1407.410668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1412.039429] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1420.037341] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1427.891391] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1435.097560] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1435.484056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1440.626576] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1442.607107] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1450.550969] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1459.184793] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1462.497609] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1474.323879] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2217h29m22s [ 1476.166800] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1478.631929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.635493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.652589] Lustre: Skipped 2 previous similar messages [ 1478.677308] Lustre: Skipped 2 previous similar messages [ 1479.717516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1479.722560] Lustre: Skipped 2 previous similar messages [ 1483.123338] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 1484.989984] Lustre: server umount lustre-MDT0000 complete [ 1485.805767] LustreError: 50349:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1485.845407] LustreError: 50349:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1488.459426] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1490.257337] LustreError: 50346:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.612774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.808288] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1495.202900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1495.299295] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1496.241578] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1499.080990] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1502.283897] Lustre: DEBUG MARKER: Wait for df (36739) ... [ 1503.707671] Lustre: DEBUG MARKER: done [ 1509.344272] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1509.347358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1509.353388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1509.356988] Lustre: Skipped 1 previous similar message [ 1510.371386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1510.383196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1517.027525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1517.035142] Lustre: Skipped 4 previous similar messages [ 1522.146937] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1522.327244] Lustre: server umount lustre-OST0000 complete [ 1527.265827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1527.271714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1527.276748] Lustre: Skipped 1 previous similar message [ 1527.283774] Lustre: Skipped 2 previous similar messages [ 1527.305728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 127.0.0.2@tcp failed: rc = -107 [ 1532.408627] Lustre: server umount lustre-MDT0000 complete [ 1533.409181] LustreError: 51077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.434839] LustreError: 51077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1535.748628] LustreError: 50331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434225 with bad export cookie 5280190246547165989 [ 1535.752294] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1535.761432] LustreError: 50331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1536.048201] Lustre: server umount lustre-MDT0001 complete [ 1546.793219] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1549.993093] Key type lgssc unregistered [ 1550.489163] LNet: 55276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1550.504654] LNetError: 55276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1550.546230] LNet: Removed LNI 192.168.203.115@tcp [ 1551.527871] Key type .llcrypt unregistered [ 1551.535874] Key type ._llcrypt unregistered [ 1591.535575] Key type ._llcrypt registered [ 1591.538256] Key type .llcrypt registered [ 1591.638239] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1592.292359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1592.326641] alg: No test for adler32 (adler32-zlib) [ 1593.319929] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 1593.544731] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1595.175994] Key type lgssc registered [ 1596.151894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1602.316316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1602.329149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1603.480675] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1603.504325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1603.653566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.010588] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1614.302923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1614.431409] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1614.460713] Lustre: 57485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1614.764705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1619.466904] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1626.892969] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1634.126370] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1641.629694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1641.763142] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1641.907594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1648.803190] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1650.332413] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1655.952046] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1664.690491] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2217h32m31s [ 1667.359771] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1677.089811] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1680.537394] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1680.631844] LustreError: 57490:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1700.746366] Lustre: lustre-MDT0000: Client 4ecf6b4b-fa4e-457b-b6e4-48cbab3ab0ca (at 192.168.203.15@tcp) reconnecting [ 1720.735646] LustreError: 57490:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1722.705837] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1724.683983] Lustre: DEBUG MARKER: done [ 1732.069188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1732.073801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.084092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1732.585273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1732.606253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1732.636868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1733.709677] Lustre: server umount lustre-OST0000 complete [ 1742.818121] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1742.836463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.860800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1742.889511] Lustre: Skipped 1 previous similar message [ 1745.906565] Lustre: server umount lustre-MDT0000 complete [ 1750.607092] LustreError: 57478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434440 with bad export cookie 4325835828624449532 [ 1750.623432] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1750.627855] LustreError: 57478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1751.408663] Lustre: server umount lustre-MDT0001 complete [ 1760.973278] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 1763.641231] Key type lgssc unregistered [ 1763.992904] LNet: 61819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.013885] LNetError: 61819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.033987] LNet: Removed LNI 192.168.203.115@tcp [ 1764.944878] Key type .llcrypt unregistered [ 1764.948251] Key type ._llcrypt unregistered [ 1802.401277] Key type ._llcrypt registered [ 1802.404397] Key type .llcrypt registered [ 1802.490854] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 1803.700485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1803.731934] alg: No test for adler32 (adler32-zlib) [ 1804.929593] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 1805.218259] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 1806.887229] Key type lgssc registered [ 1808.002378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1810.909768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1810.932348] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1812.108960] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1812.129174] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1812.221291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1812.928145] Lustre: Failing over lustre-MDT0000 [ 1813.135607] Lustre: server umount lustre-MDT0000 complete [ 1817.554812] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:35:05 (1787434505) [ 1821.967481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.252330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.170438] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1831.209127] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.295399] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1831.326433] Lustre: 64596:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1831.522159] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.657934] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1839.852956] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1844.143209] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1849.181412] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1849.347893] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1849.521509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.300911] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1857.161054] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1859.901845] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1866.049817] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1872.358041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1872.375823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.398061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.985281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.993668] Lustre: Skipped 4 previous similar messages [ 1878.806663] Lustre: server umount lustre-MDT0000 complete [ 1882.103871] LustreError: 64586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434571 with bad export cookie 13142096990821335027 [ 1882.104909] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1882.124422] LustreError: 64586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1882.476347] Lustre: server umount lustre-MDT0001 complete [ 1886.007417] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1890.357661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1898.465041] Lustre: 63519:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787434572/real 1787434572] req@000000005229a627 x1874260913066112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787434588 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1898.492657] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.509876] Lustre: Skipped 2 previous similar messages [ 1899.221258] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1903.583371] Lustre: 63521:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787434577/real 1787434577] req@00000000ea3480e6 x1874260913066240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787434593 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1904.608948] Lustre: 63522:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787434578/real 1787434578] req@000000000e8386bc x1874260913066368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787434594 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1907.743980] LustreError: 63518:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005a3f97d5 x1874260913067136/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1908.232268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.339217] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1909.215790] Lustre: 63520:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787434582/real 1787434582] req@00000000bec8d1f3 x1874260913066624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787434598 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1912.323864] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1913.376540] Lustre: 63519:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787434587/real 1787434587] req@00000000b8cd5704 x1874260913066752/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787434603 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1914.337623] LustreError: 68365:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.354894] LustreError: 68365:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 1919.460372] LustreError: 68364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1919.477325] LustreError: 68364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1920.059435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1920.377516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1924.494306] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1925.607304] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1927.659414] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1932.319270] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1938.121416] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1945.924612] Lustre: server umount lustre-MDT0000 complete [ 1946.083253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1946.094181] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.118579] LustreError: 66442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.130358] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1948.133885] LustreError: 66441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1948.148805] Lustre: Skipped 1 previous similar message [ 1948.183154] LustreError: 66441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1949.097360] LustreError: 64586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434638 with bad export cookie 13142096990821337302 [ 1949.102748] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1949.332384] Lustre: server umount lustre-MDT0001 complete [ 1955.368370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1963.865240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.631717] Lustre: 63522:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787434643/real 1787434643] req@0000000095af3f59 x1874260913093248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787434659 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.670849] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.793664] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb6621b9555a2ce39 [ 1973.806090] Lustre: MGC192.168.203.115@tcp: Connection restored to 0@lo (at 0@lo) [ 1974.076517] LustreError: 69110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1974.135525] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1974.203953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1978.326322] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1984.480822] LustreError: 71046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.499224] LustreError: 71046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1985.492895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.909310] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1989.617539] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 1996.369637] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2002.422729] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2009.575712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.581936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.591520] Lustre: Skipped 2 previous similar messages [ 2009.620055] Lustre: Skipped 2 previous similar messages [ 2014.689063] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.698973] Lustre: Skipped 2 previous similar messages [ 2015.443308] Lustre: server umount lustre-MDT0000 complete [ 2019.085988] LustreError: 64588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434708 with bad export cookie 13142096990821338681 [ 2019.087888] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2019.105210] LustreError: 64588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2019.424464] Lustre: server umount lustre-MDT0001 complete [ 2024.711744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2027.964105] Lustre: DEBUG MARKER: files compared the same [ 2040.036603] Lustre: server umount lustre-OST0000 complete [ 2057.842535] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2060.469662] Key type lgssc unregistered [ 2060.749737] LNet: 74328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2060.761657] LNetError: 74328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2060.787664] LNet: Removed LNI 192.168.203.115@tcp [ 2062.031229] Key type .llcrypt unregistered [ 2062.033086] Key type ._llcrypt unregistered [ 2089.022884] Key type ._llcrypt registered [ 2089.026692] Key type .llcrypt registered [ 2089.200314] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2090.784339] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2090.853934] alg: No test for adler32 (adler32-zlib) [ 2091.899570] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 2092.158275] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2093.888099] Key type lgssc registered [ 2095.683785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2105.924531] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2105.946976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2107.582697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2111.987293] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all [ 2118.122911] LustreError: 75745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2118.152043] LustreError: 75745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2122.314285] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.729805] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2128.664149] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all [ 2136.964855] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2142.407961] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2149.215985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2149.562361] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2152.627129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2155.518880] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug +malloc all [ 2163.126983] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2178.532448] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2178.544883] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2178.581682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2179.783953] Lustre: server umount lustre-OST0000 complete [ 2188.768630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.779047] Lustre: Skipped 1 previous similar message [ 2188.789898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2188.801564] Lustre: Skipped 2 previous similar messages [ 2191.088193] Lustre: server umount lustre-MDT0000 complete [ 2193.893159] LustreError: 75746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.916474] LustreError: 75746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2195.303943] LustreError: 75728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787434885 with bad export cookie 3975567765637637184 [ 2195.309871] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.316966] LustreError: 75728:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2195.829465] Lustre: server umount lustre-MDT0001 complete [ 2207.866213] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2210.963516] Key type lgssc unregistered [ 2211.252457] LNet: 79284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2211.262437] LNetError: 79284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2211.275208] LNet: Removed LNI 192.168.203.115@tcp [ 2212.192873] Key type .llcrypt unregistered [ 2212.206520] Key type ._llcrypt unregistered [ 2235.869883] Key type ._llcrypt registered [ 2235.871525] Key type .llcrypt registered [ 2235.979431] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2237.358977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2237.403032] alg: No test for adler32 (adler32-zlib) [ 2238.562264] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 2238.915814] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2240.567201] Key type lgssc registered [ 2241.823974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2250.090262] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2250.114164] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2265.631403] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2270.751321] LustreError: 80673:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [ 2297.375180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.181430] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2310.632995] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2317.887888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.034956] Lustre: *** cfs_fail_loc=706, val=0*** [ 2318.290408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2318.360489] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2322.537490] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2323.430237] LustreError: 80699:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.453241] LustreError: 80699:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2328.547290] LustreError: 80698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2331.274059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.676198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.285633] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2345.784466] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2352.572850] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2357.217831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2357.233902] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2357.264055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2358.257321] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2358.264501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2361.250678] Lustre: server umount lustre-OST0000 complete [ 2367.457482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2367.473666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2367.508637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2367.531374] Lustre: Skipped 1 previous similar message [ 2373.608285] LustreError: 81759:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.633231] LustreError: 81759:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2373.689396] Lustre: server umount lustre-MDT0000 complete [ 2377.367714] LustreError: 80680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435067 with bad export cookie 9899852558892511992 [ 2377.383536] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2377.676961] Lustre: server umount lustre-MDT0001 complete [ 2388.774529] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2391.105360] Key type lgssc unregistered [ 2391.454661] LNet: 84293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2391.459640] LNetError: 84293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.479412] LNet: Removed LNI 192.168.203.115@tcp [ 2392.215389] Key type .llcrypt unregistered [ 2392.219625] Key type ._llcrypt unregistered [ 2419.919076] Key type ._llcrypt registered [ 2419.922964] Key type .llcrypt registered [ 2420.047253] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2421.422970] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2421.606249] alg: No test for adler32 (adler32-zlib) [ 2422.820669] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 2423.236513] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2424.967552] Key type lgssc registered [ 2426.398447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2434.315779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2434.359731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.547392] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2447.401713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.783509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2452.250529] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2452.960219] LustreError: 86399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2452.972449] LustreError: 86399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2458.081112] LustreError: 86402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.110570] LustreError: 86402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2460.027943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2460.447587] LustreError: 86399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2460.476245] LustreError: 86399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2460.507526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2465.768325] LustreError: 86397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.808365] LustreError: 86397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2467.524673] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2470.884525] LustreError: 86397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2470.901834] LustreError: 86397:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2476.931068] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2476.969787] Lustre: MGS: Client b3be686c-5fa8-4085-9a35-4cfc055921f7 (at 0@lo) reconnecting [ 2477.014337] LustreError: 86413:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2477.330744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2477.468597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2482.814487] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2498.024804] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2498.026063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2498.035028] Lustre: Skipped 1 previous similar message [ 2498.043102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2498.060112] Lustre: Skipped 1 previous similar message [ 2503.150417] LustreError: 87125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.166531] LustreError: 87125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2503.246371] Lustre: server umount lustre-OST0000 complete [ 2508.257225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2508.259834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2508.279557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2508.290853] Lustre: Skipped 1 previous similar message [ 2513.378233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.394606] Lustre: server umount lustre-MDT0000 complete [ 2517.468973] Lustre: server umount lustre-MDT0001 complete [ 2521.405278] Lustre: server umount MGS complete [ 2533.448025] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:47:01 (1787435221) [ 2564.572802] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 2578.812423] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2616.244599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.049279] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2632.056109] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2632.379759] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2632.441427] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2632.513623] Lustre: lustre-MDT0001: new disk, initializing [ 2632.585119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.605577] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2632.609851] Lustre: 93571:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2632.615262] LustreError: 93571:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2638.244419] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2651.261499] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2651.581354] Lustre: lustre-OST0000: new disk, initializing [ 2651.594440] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2651.612906] Lustre: 94337:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2651.765326] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2658.919150] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2659.367211] Lustre: 94637:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000f4e61497 x1874261561089536/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787435365 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2670.228228] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2677.714813] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2677.730267] Lustre: MGS: Client 48be2540-0292-41db-9ec2-7c985ee82eec (at 0@lo) reconnecting [ 2677.804302] Lustre: 92837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2677.824880] LustreError: 93557:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2677.881235] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2678.080264] Lustre: lustre-MDT0000: new disk, initializing [ 2678.235302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.282806] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2678.322390] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2679.401940] Lustre: 95407:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000a4a9d8da x1874261561096576/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787435385 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2683.646684] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2684.514041] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2686.958040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2686.977005] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2688.499120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2688.516484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2688.566261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2714.592506] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2714.603288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.623603] Lustre: Skipped 1 previous similar message [ 2714.639161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2719.719887] Lustre: server umount lustre-OST0000 complete [ 2725.859570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.872097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.883901] Lustre: Skipped 2 previous similar messages [ 2725.912863] Lustre: Skipped 3 previous similar messages [ 2730.435321] Lustre: server umount lustre-MDT0000 complete [ 2730.980719] LustreError: 93546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2730.989175] LustreError: 93546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2741.847385] Lustre: server umount lustre-MDT0001 complete [ 2746.191921] Lustre: server umount MGS complete [ 2758.668643] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:50:46 (1787435446) [ 2780.351699] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2783.723415] Key type lgssc unregistered [ 2784.082800] LNet: 97909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2784.091932] LNetError: 97909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2784.105527] LNet: Removed LNI 192.168.203.115@tcp [ 2784.879073] Key type .llcrypt unregistered [ 2784.880867] Key type ._llcrypt unregistered [ 2805.526129] Key type ._llcrypt registered [ 2805.527930] Key type .llcrypt registered [ 2805.702199] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2807.212454] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2807.270357] alg: No test for adler32 (adler32-zlib) [ 2808.338752] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 2808.621565] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2810.327154] Key type lgssc registered [ 2811.707605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2821.544393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2821.590916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2823.225465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2828.360334] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2833.391708] LustreError: 99133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2833.414712] LustreError: 99133:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2838.501909] LustreError: 99134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2838.858969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.322293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2846.728579] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2858.274827] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2867.232705] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2876.781056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2877.202170] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2884.015666] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2892.674460] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2893.616581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2909.164178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.166124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2909.208309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.214752] Lustre: Skipped 1 previous similar message [ 2912.702812] Lustre: server umount lustre-OST0000 complete [ 2919.393185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.418168] Lustre: Skipped 1 previous similar message [ 2919.425482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.432427] Lustre: Skipped 1 previous similar message [ 2923.545593] Lustre: server umount lustre-MDT0000 complete [ 2924.512302] LustreError: 99133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2927.686560] LustreError: 100979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435617 with bad export cookie 9912886393840774119 [ 2927.695556] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2927.713612] LustreError: 100979:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2928.318399] Lustre: server umount lustre-MDT0001 complete [ 2940.243684] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 2943.557131] Key type lgssc unregistered [ 2943.950772] LNet: 102692:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2943.961978] LNetError: 102692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2943.996681] LNet: Removed LNI 192.168.203.115@tcp [ 2945.058192] Key type .llcrypt unregistered [ 2945.060537] Key type ._llcrypt unregistered [ 2961.676611] Key type ._llcrypt registered [ 2961.678704] Key type .llcrypt registered [ 2961.784536] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 2962.877143] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2962.984957] alg: No test for adler32 (adler32-zlib) [ 2964.343740] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 2964.675156] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 2966.335822] Key type lgssc registered [ 2967.505313] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2973.619178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2973.629099] LustreError: 103761:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2973.834772] LustreError: 103777:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2973.843901] LustreError: 103761:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4806 [ 2973.855349] LustreError: 103761:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2973.871722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2973.873490] LustreError: 103761:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2975.528585] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2983.356877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2983.643539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2988.415802] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 2994.181567] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing lsmod [ 2996.908724] LustreError: 104981:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2996.953948] LustreError: 104985:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2996.966355] LustreError: 104981:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4952 [ 2996.979207] LustreError: 104985:0:(obd_mount.c:234:lustre_start_simple()) lustre-OST0000-osd attach error -17 [ 2996.988651] LustreError: 104985:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -17 [ 2997.006348] LustreError: 104985:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -17 [ 2997.018427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2997.527063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.354908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3000.360775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3000.372464] Lustre: Skipped 1 previous similar message [ 3000.391478] Lustre: Skipped 1 previous similar message [ 3005.411061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3005.418763] Lustre: Skipped 3 previous similar messages [ 3006.863913] Lustre: server umount lustre-MDT0000 complete [ 3008.484851] LustreError: 103814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3008.497836] LustreError: 103814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3010.048819] LustreError: 103795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435699 with bad export cookie 6953459937972006930 [ 3010.058721] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3010.064171] LustreError: 103795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3010.568946] LustreError: 104246:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000494ecae9 x1874262128348416/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3010.753846] Lustre: server umount lustre-MDT0001 complete [ 3025.111715] Lustre: server umount lustre-OST0000 complete [ 3034.487535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3034.943132] LustreError: 106050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3035.054170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3040.227576] LustreError: 106050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3040.264057] LustreError: 106050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3040.317821] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3044.324067] LustreError: 106075:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3049.444837] LustreError: 106050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3050.862385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3051.391423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3057.538506] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3066.374414] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3072.878751] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3080.429820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3080.873399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3087.692530] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3095.957909] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3104.428977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3114.978254] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3114.985574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3114.990654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3115.028098] Lustre: Skipped 2 previous similar messages [ 3119.196407] Lustre: server umount lustre-OST0000 complete [ 3123.168211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3123.182426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3123.206164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3123.210607] Lustre: Skipped 1 previous similar message [ 3125.217326] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.435670] Lustre: server umount lustre-MDT0000 complete [ 3130.337828] LustreError: 106047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3130.363443] LustreError: 106047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3133.244219] LustreError: 106031:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435823 with bad export cookie 6953459937972007623 [ 3133.261430] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3133.550818] Lustre: server umount lustre-MDT0001 complete [ 3145.378499] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3148.391808] Key type lgssc unregistered [ 3148.726922] LNet: 109607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3148.743556] LNetError: 109607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3149.797832] LNet: Removed LNI 192.168.203.115@tcp [ 3150.853226] Key type .llcrypt unregistered [ 3150.857681] Key type ._llcrypt unregistered [ 3176.141876] Key type ._llcrypt registered [ 3176.143936] Key type .llcrypt registered [ 3176.253760] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3177.609592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3177.648905] alg: No test for adler32 (adler32-zlib) [ 3178.994494] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 3179.352066] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3181.087429] Key type lgssc registered [ 3182.516832] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3191.265187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3191.315738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3192.814774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3198.459743] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3203.042529] LustreError: 111038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3203.075272] LustreError: 111038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3208.164081] LustreError: 111039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3208.244686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3208.758677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3212.861390] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3220.277123] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3226.399491] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3233.599572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3233.962752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3238.080320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3239.952691] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3247.764891] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3257.583443] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3263.003593] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3269.089777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3269.106172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.119861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3269.130220] Lustre: Skipped 1 previous similar message [ 3269.149197] Lustre: Skipped 1 previous similar message [ 3273.775894] Lustre: server umount lustre-OST0000 complete [ 3279.329297] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.343229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.350100] Lustre: Skipped 1 previous similar message [ 3279.364019] Lustre: Skipped 1 previous similar message [ 3284.458678] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3284.467976] Lustre: Skipped 1 previous similar message [ 3284.661405] Lustre: server umount lustre-MDT0000 complete [ 3289.571781] LustreError: 111039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3289.612277] LustreError: 111039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3290.549630] LustreError: 111020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787435980 with bad export cookie 335926795829879244 [ 3290.565239] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3290.590632] LustreError: 111020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3291.470577] Lustre: server umount lustre-MDT0001 complete [ 3306.935878] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3309.622245] Key type lgssc unregistered [ 3309.884259] LNet: 114682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3309.897616] LNetError: 114682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3309.909058] LNet: Removed LNI 192.168.203.115@tcp [ 3310.881079] Key type .llcrypt unregistered [ 3310.884352] Key type ._llcrypt unregistered [ 3334.071854] Key type ._llcrypt registered [ 3334.073477] Key type .llcrypt registered [ 3334.228936] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3335.546070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3335.751056] alg: No test for adler32 (adler32-zlib) [ 3336.928227] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 3337.184391] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3338.855175] Key type lgssc registered [ 3340.253750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3348.698029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3348.760944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.114676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3353.836376] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3359.207411] LustreError: 115905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3359.235495] LustreError: 115905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3363.750964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.472111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3370.014414] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3380.689771] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3386.407904] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3393.839627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3394.300325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3395.387403] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3400.623388] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3409.686506] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3420.130505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3420.146282] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3420.153225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3420.175281] Lustre: Skipped 1 previous similar message [ 3425.103844] Lustre: server umount lustre-OST0000 complete [ 3430.388523] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.390294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3430.400811] Lustre: Skipped 1 previous similar message [ 3430.418238] Lustre: Skipped 2 previous similar messages [ 3435.489319] LustreError: 118398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.521342] LustreError: 118398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3437.520939] Lustre: server umount lustre-MDT0000 complete [ 3440.609988] LustreError: 115906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3440.639696] LustreError: 115906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3441.472197] LustreError: 115885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436131 with bad export cookie 17335651370950243088 [ 3441.478445] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3441.496369] LustreError: 115885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3441.837176] Lustre: server umount lustre-MDT0001 complete [ 3455.933337] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3459.758926] Key type lgssc unregistered [ 3460.195433] LNet: 119455:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3460.211111] LNetError: 119455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3460.234200] LNet: Removed LNI 192.168.203.115@tcp [ 3460.921073] Key type .llcrypt unregistered [ 3460.926069] Key type ._llcrypt unregistered [ 3490.244761] Key type ._llcrypt registered [ 3490.246751] Key type .llcrypt registered [ 3490.388698] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3492.109935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3492.156788] alg: No test for adler32 (adler32-zlib) [ 3493.661109] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 3494.077776] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3495.735310] Key type lgssc registered [ 3498.229326] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3507.932835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3507.950739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3509.400575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.340712] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3519.977371] LustreError: 120876:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3520.004388] LustreError: 120876:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3522.755757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3523.132738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3527.922961] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3535.967863] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3541.815565] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3548.911389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3549.302446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3554.475454] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3556.214606] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3565.737767] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3574.820437] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3579.019562] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3579.025645] Lustre: Skipped 2 previous similar messages [ 3583.056834] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3583.090321] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3583.811794] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3583.823958] Lustre: Skipped 2 previous similar messages [ 3593.294253] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3593.298495] Lustre: Skipped 1 previous similar message [ 3599.725996] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3599.735964] Lustre: Skipped 2 previous similar messages [ 3602.231851] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.15@tcp 192.168.0.[2,10]@tcp [ 3602.259571] Lustre: Skipped 1 previous similar message [ 3607.514258] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.15/24@tcp in log lustre-MDT0000 [ 3607.526814] Lustre: Skipped 2 previous similar messages [ 3611.343821] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.15/24@tcp [ 3618.796161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3618.805948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3618.809116] LustreError: Skipped 1 previous similar message [ 3618.824248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3619.301489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3619.313066] LustreError: Skipped 3 previous similar messages [ 3624.430058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3624.449322] Lustre: Skipped 3 previous similar messages [ 3624.903344] Lustre: server umount lustre-OST0000 complete [ 3634.659617] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3634.670162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3634.678734] Lustre: Skipped 2 previous similar messages [ 3634.696015] Lustre: Skipped 1 previous similar message [ 3638.346444] Lustre: server umount lustre-MDT0000 complete [ 3639.777616] LustreError: 121873:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3639.792197] LustreError: 121873:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3642.273658] LustreError: 122719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436332 with bad export cookie 5824500142096708260 [ 3642.277358] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3642.284778] LustreError: 122719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3642.716214] Lustre: server umount lustre-MDT0001 complete [ 3655.066613] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3657.905657] Key type lgssc unregistered [ 3658.134606] LNet: 125825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3658.142854] LNetError: 125825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3658.163280] LNet: Removed LNI 192.168.203.115@tcp [ 3658.971504] Key type .llcrypt unregistered [ 3658.973651] Key type ._llcrypt unregistered [ 3689.769215] Key type ._llcrypt registered [ 3689.772652] Key type .llcrypt registered [ 3689.875115] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3690.849607] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3691.016044] alg: No test for adler32 (adler32-zlib) [ 3692.140714] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 3692.434291] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3694.119162] Key type lgssc registered [ 3695.050056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3702.347807] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3702.367712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3703.877906] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3703.921779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3703.922584] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.15/24@tcp [ 3708.517871] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3713.516284] LustreError: 127391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3713.547762] LustreError: 127391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3717.861447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.123035] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3718.157279] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.15/24@tcp [ 3718.193384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3722.771783] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3730.368527] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3735.952167] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3741.666102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3741.965186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3748.168612] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3748.552860] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3756.677408] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3768.812167] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3768.820370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3768.823498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3768.838729] Lustre: Skipped 1 previous similar message [ 3768.860778] Lustre: Skipped 1 previous similar message [ 3773.224836] Lustre: server umount lustre-OST0000 complete [ 3779.051321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3779.063416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3779.091486] Lustre: Skipped 1 previous similar message [ 3779.130966] Lustre: Skipped 1 previous similar message [ 3782.957541] Lustre: server umount lustre-MDT0000 complete [ 3784.160511] LustreError: 127418:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3784.167789] LustreError: 127418:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3786.423706] LustreError: 130025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436476 with bad export cookie 3781419861262374106 [ 3786.425476] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3786.448021] LustreError: 130025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3786.956664] Lustre: server umount lustre-MDT0001 complete [ 3803.531504] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 3807.260847] Key type lgssc unregistered [ 3807.558504] LNet: 130983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3807.588855] LNetError: 130983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3807.643968] LNet: Removed LNI 192.168.203.115@tcp [ 3809.334933] Key type .llcrypt unregistered [ 3809.347182] Key type ._llcrypt unregistered [ 3847.075657] Key type ._llcrypt registered [ 3847.079542] Key type .llcrypt registered [ 3847.227139] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 3869.667825] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 3870.806374] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3870.863067] alg: No test for adler32 (adler32-zlib) [ 3871.983072] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 3872.401234] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 3874.263409] Key type lgssc registered [ 3875.351609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3915.968880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3916.037113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3917.448385] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3917.479295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3917.558249] Lustre: lustre-MDT0000: new disk, initializing [ 3917.664963] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3917.679310] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3924.003907] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3936.169652] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3936.382755] Lustre: 134386:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3936.503354] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3936.511139] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3936.513624] Lustre: Skipped 1 previous similar message [ 3936.668845] Lustre: lustre-MDT0001: new disk, initializing [ 3936.820906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3936.897067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3936.913356] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3942.043089] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3951.516767] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3957.171084] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3964.680348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3964.978334] Lustre: lustre-OST0000: new disk, initializing [ 3964.982599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3964.994286] Lustre: 136356:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3965.080343] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3970.161528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3970.183954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3970.386531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3974.155648] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 3986.809411] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3992.195333] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3992.383775] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3996.846608] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3997.213783] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4001.251733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4001.254525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4001.263205] LustreError: Skipped 1 previous similar message [ 4001.269176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4001.288800] Lustre: Skipped 1 previous similar message [ 4001.317882] Lustre: Skipped 1 previous similar message [ 4005.443176] Lustre: server umount lustre-OST0000 complete [ 4011.492608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4011.496068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4011.510420] Lustre: Skipped 1 previous similar message [ 4015.887882] Lustre: server umount lustre-MDT0000 complete [ 4020.369792] LustreError: 134376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436710 with bad export cookie 16968850583865349587 [ 4020.385850] LustreError: 134376:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4020.388465] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4020.680834] Lustre: server umount lustre-MDT0001 complete [ 4030.513895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4031.010161] LustreError: 138575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4031.024231] LustreError: 138575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4031.062251] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4035.620234] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4036.065243] LustreError: 138576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4040.161516] LustreError: 138575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4045.296970] LustreError: 138576:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4046.144126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4046.421442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4052.060463] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4062.915832] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4070.919344] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4078.598318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4078.961147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4085.279705] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4093.922743] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4099.770919] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4101.145682] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 4109.063889] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4109.375846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4119.358065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4119.740669] Lustre: lustre-OST0001: new disk, initializing [ 4119.747632] Lustre: lustre-OST0001: Not available for connect from 192.168.203.15@tcp (not set up) [ 4119.753426] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4119.767869] Lustre: Skipped 1 previous similar message [ 4119.798607] Lustre: 141956:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4119.923135] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4127.938590] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4128.344814] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4128.363815] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4128.450110] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4136.905341] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4137.371202] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4143.305563] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4143.639517] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4152.015175] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99d1984f9800.ost_server_uuid 50 [ 4154.375412] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99d1984f9800.ost_server_uuid in IDLE state after 0 sec [ 4169.703046] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4169.707887] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4169.721230] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4169.725666] Lustre: Skipped 1 previous similar message [ 4172.209077] Lustre: server umount lustre-OST0001 complete [ 4174.818303] LustreError: 140398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.845971] LustreError: 140398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4177.386268] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4177.407527] LustreError: Skipped 1 previous similar message [ 4177.426176] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4177.449164] Lustre: Skipped 1 previous similar message [ 4178.409273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4178.418608] Lustre: Skipped 1 previous similar message [ 4185.090190] LustreError: 140399:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4185.116296] LustreError: 140399:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4191.200322] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4191.400554] Lustre: server umount lustre-OST0000 complete [ 4199.917108] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4199.936392] LustreError: Skipped 1 previous similar message [ 4199.943669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4199.958410] Lustre: Skipped 1 previous similar message [ 4199.963970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4199.973763] Lustre: Skipped 6 previous similar messages [ 4210.143752] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4210.577406] Lustre: server umount lustre-MDT0000 complete [ 4210.673216] LustreError: 139303:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4210.725600] LustreError: 139303:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4215.278780] LustreError: 138556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787436905 with bad export cookie 16968850583865350462 [ 4215.299532] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4215.305813] LustreError: 138556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4215.877455] Lustre: server umount lustre-MDT0001 complete [ 4241.708752] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4246.314515] Key type lgssc unregistered [ 4246.640688] LNet: 144969:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4246.649505] LNetError: 144969:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4247.717634] LNet: Removed LNI 192.168.203.115@tcp [ 4248.803223] Key type .llcrypt unregistered [ 4248.806186] Key type ._llcrypt unregistered [ 4294.038377] Key type ._llcrypt registered [ 4294.040673] Key type .llcrypt registered [ 4294.152831] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4312.656570] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4314.213886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4314.236194] alg: No test for adler32 (adler32-zlib) [ 4315.270811] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 4315.497218] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4317.247197] Key type lgssc registered [ 4319.094822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4358.678149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4358.715779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.951777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4360.020673] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4360.133784] Lustre: lustre-MDT0000: new disk, initializing [ 4360.211897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4360.245103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4365.238070] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4377.177881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.328793] Lustre: 149129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4377.382732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4377.387861] Lustre: Skipped 1 previous similar message [ 4377.513272] Lustre: lustre-MDT0001: new disk, initializing [ 4377.657517] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4377.691781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4377.702259] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4382.204568] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4393.918072] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4400.191813] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4407.704845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4407.963623] Lustre: lustre-OST0000: new disk, initializing [ 4407.969201] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4407.984077] Lustre: 151098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4408.091747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4414.429665] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4417.194884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4417.203313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4417.423918] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4424.683202] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4429.669068] Lustre: Failing over lustre-OST0000 [ 4429.800424] Lustre: server umount lustre-OST0000 complete [ 4446.175276] Lustre: 147348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787437120/real 1787437120] req@00000000163e9c02 x1874263544989184/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787437136 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4446.228859] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4448.225735] Lustre: 147349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787437122/real 1787437122] req@00000000b216c815 x1874263544990208/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787437138 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4448.288319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4450.228714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4450.694040] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4450.737326] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4453.727114] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787437127/real 1787437127] req@00000000b1d01d79 x1874263544991360/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787437143 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4453.776539] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4456.807428] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4458.848914] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787437132/real 1787437132] req@00000000b91a5eff x1874263544992512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787437148 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4458.869723] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4458.885181] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4458.969582] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4458.969809] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4462.955389] Lustre: Failing over lustre-MDT0000 [ 4463.164375] Lustre: server umount lustre-MDT0000 complete [ 4463.967180] Lustre: 147350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787437136/real 1787437136] req@0000000042b2967d x1874263544993536/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787437152 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4464.610123] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4464.617644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.572369] LustreError: 149929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.601898] LustreError: 149929:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4470.241448] LustreError: 149140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4470.248898] LustreError: 149140:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4473.682671] LustreError: 149136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.809279] LustreError: 149136:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.864255] LustreError: 149136:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4480.479168] Lustre: 147349:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787437154/real 1787437154] req@00000000bec8d1f3 x1874263545006720/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787437170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4480.516680] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4483.931024] LustreError: 149135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.009443] LustreError: 149135:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4485.770872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4490.722864] LustreError: 147347:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000051e803e6 x1874263545012096/t0(0) o250->MGC192.168.203.115@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4491.208588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4491.306973] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4494.159859] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4496.381777] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4496.388446] Lustre: Skipped 1 previous similar message [ 4496.434356] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4496.501365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4498.063289] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4511.723244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4511.760631] Lustre: Skipped 3 previous similar messages [ 4511.764779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4511.790203] Lustre: Skipped 1 previous similar message [ 4516.843984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4516.852119] Lustre: Skipped 1 previous similar message [ 4517.873669] Lustre: server umount lustre-OST0000 complete [ 4527.076542] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4527.080967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4527.110467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4527.121436] Lustre: Skipped 1 previous similar message [ 4530.181130] Lustre: server umount lustre-MDT0000 complete [ 4532.195627] LustreError: 152020:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4532.212612] LustreError: 152020:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4534.183632] LustreError: 149121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437224 with bad export cookie 3752177897735003963 [ 4534.205047] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4534.205208] LustreError: 149121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4534.514640] Lustre: server umount lustre-MDT0001 complete [ 4547.480602] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 4550.908875] Key type lgssc unregistered [ 4551.369169] LNet: 154766:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4551.379112] LNetError: 154766:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4551.398686] LNet: Removed LNI 192.168.203.115@tcp [ 4552.426877] Key type .llcrypt unregistered [ 4552.436309] Key type ._llcrypt unregistered [ 4592.931797] Key type ._llcrypt registered [ 4592.933840] Key type .llcrypt registered [ 4592.995506] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4593.692441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4593.713337] alg: No test for adler32 (adler32-zlib) [ 4594.701259] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 4594.909675] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 4596.535187] Key type lgssc registered [ 4597.560400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4605.882519] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4605.952898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.530808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4612.910400] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4617.698968] LustreError: 156287:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4617.734202] LustreError: 156287:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4622.562604] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4622.816626] LustreError: 156286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.455619] LustreError: 156315:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4626.464322] LustreError: 156315:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4626.480201] LustreError: 156315:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4626.814881] Lustre: server umount lustre-MDT0000 complete [ 4644.523585] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 4654.794925] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 4688.998658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4689.256985] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4689.311288] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4689.429624] Lustre: lustre-MDT0000: new disk, initializing [ 4689.658481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.682498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4694.483620] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4707.527037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.731036] Lustre: 159620:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4707.763684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4707.766486] Lustre: Skipped 1 previous similar message [ 4707.831953] Lustre: lustre-MDT0001: new disk, initializing [ 4707.988298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4708.049192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4708.076655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4715.263984] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4728.455612] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4735.612629] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4745.884325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4746.378947] Lustre: lustre-OST0000: new disk, initializing [ 4746.385583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4746.412696] Lustre: 161588:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4746.644468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4752.313939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4752.336103] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4752.380449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 4757.153153] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 4772.612831] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4778.053308] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4780.985670] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:24:28 (1787437468) [ 5044.192388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.198635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.210022] Lustre: Skipped 2 previous similar messages [ 5044.219573] Lustre: Skipped 1 previous similar message [ 5047.291488] Lustre: server umount lustre-MDT0000 complete [ 5049.314823] LustreError: 159630:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.333651] LustreError: 159630:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5050.301379] LustreError: 159612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437740 with bad export cookie 2467164377180994975 [ 5050.304389] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5050.311811] LustreError: 159612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5054.431747] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5054.436806] LustreError: 161757:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5054.449719] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5054.471556] Lustre: Skipped 1 previous similar message [ 5055.457887] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5056.770224] Lustre: server umount lustre-MDT0001 complete [ 5066.724300] Lustre: server umount lustre-OST0000 complete [ 5076.035747] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5085.453440] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5125.619621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.970453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5126.033561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5126.171779] Lustre: lustre-MDT0000: new disk, initializing [ 5126.384817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5126.427296] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5131.235697] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5143.489158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5143.607857] Lustre: 166256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5143.665802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5143.675129] Lustre: Skipped 1 previous similar message [ 5143.739803] Lustre: lustre-MDT0001: new disk, initializing [ 5143.889204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5143.941878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5143.960814] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5148.743049] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5158.860412] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5164.624361] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5170.274044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5170.551840] Lustre: lustre-OST0000: new disk, initializing [ 5170.555704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5170.564134] Lustre: 168226:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5170.637090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.480768] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5178.433406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5178.444810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5178.561418] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5183.500802] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5187.428213] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5187.583090] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5191.378302] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5191.555923] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5194.216529] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5194.221559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.227412] LustreError: Skipped 1 previous similar message [ 5194.228765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5194.240340] Lustre: Skipped 1 previous similar message [ 5199.292368] Lustre: server umount lustre-OST0000 complete [ 5204.452921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.458584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.466808] Lustre: Skipped 1 previous similar message [ 5204.478569] Lustre: Skipped 2 previous similar messages [ 5209.029561] Lustre: server umount lustre-MDT0000 complete [ 5209.568315] LustreError: 166269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5209.581395] LustreError: 166269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5212.590489] LustreError: 166251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787437902 with bad export cookie 2467164377181153805 [ 5212.612890] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5212.963915] Lustre: server umount lustre-MDT0001 complete [ 5220.086543] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:31:48 (1787437908) [ 5234.617204] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5242.933811] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5273.393690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5273.724588] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5273.775354] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5273.906408] Lustre: lustre-MDT0000: new disk, initializing [ 5274.001922] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5274.018140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5278.059988] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5286.910791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5287.017567] Lustre: 172978:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5287.245442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5290.816877] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5299.595632] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5305.365371] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5313.087812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5313.380711] Lustre: lustre-OST0000: new disk, initializing [ 5313.383789] Lustre: Skipped 1 previous similar message [ 5313.387199] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5313.394069] Lustre: Skipped 2 previous similar messages [ 5313.405775] Lustre: 174955:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5321.964454] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5322.373293] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5322.391658] Lustre: Skipped 1 previous similar message [ 5322.412469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5322.617455] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5332.164529] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5343.208536] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.210700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.214265] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5343.252021] Lustre: Skipped 1 previous similar message [ 5345.472629] Lustre: server umount lustre-OST0000 complete [ 5353.455486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.468451] Lustre: Skipped 1 previous similar message [ 5355.961857] Lustre: server umount lustre-MDT0000 complete [ 5358.560360] LustreError: 172990:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5358.591145] LustreError: 172990:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5359.807769] LustreError: 172973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438049 with bad export cookie 2467164377181154687 [ 5359.828564] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5359.830223] LustreError: 172973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 5360.181443] Lustre: server umount lustre-MDT0001 complete [ 5367.559176] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:34:15 (1787438055) [ 5386.319775] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 5396.229417] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5435.887332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5436.141624] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5436.171197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5436.231240] Lustre: lustre-MDT0000: new disk, initializing [ 5436.288205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5436.293262] Lustre: Skipped 2 previous similar messages [ 5436.307100] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5440.467442] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5450.266886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5450.406182] Lustre: 179281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5450.905208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5455.656492] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5465.334989] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5472.582963] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5479.404587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5479.705560] Lustre: 181256:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5486.214610] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5488.187363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5488.375823] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5497.416053] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5509.100757] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5509.104301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5509.124483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5509.137955] LustreError: Skipped 1 previous similar message [ 5509.160478] Lustre: Skipped 4 previous similar messages [ 5510.398037] Lustre: server umount lustre-OST0000 complete [ 5520.207228] Lustre: server umount lustre-MDT0000 complete [ 5523.621183] LustreError: 179272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438213 with bad export cookie 2467164377181155772 [ 5523.627462] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5523.630430] LustreError: 179272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5523.905622] Lustre: server umount lustre-MDT0001 complete [ 5533.904639] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5536.553164] Key type lgssc unregistered [ 5536.937886] LNet: 183117:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5536.955314] LNetError: 183117:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5536.982574] LNet: Removed LNI 192.168.203.115@tcp [ 5538.048582] Key type .llcrypt unregistered [ 5538.050987] Key type ._llcrypt unregistered [ 5568.778907] Key type ._llcrypt registered [ 5568.784711] Key type .llcrypt registered [ 5568.878246] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5569.556605] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5569.719675] alg: No test for adler32 (adler32-zlib) [ 5570.934490] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 5571.204667] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5572.919450] Key type lgssc registered [ 5573.972885] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5582.161559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5582.192476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5583.686724] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5587.729394] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5594.090896] LustreError: 184533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.112559] LustreError: 184533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5595.272867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5595.701083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5600.224129] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5607.471038] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5613.318573] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5621.972099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5622.485851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5628.666714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5629.592741] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5638.963607] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5659.629963] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5659.640360] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5659.653857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5661.666663] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5661.667052] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5661.670401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5663.688407] Lustre: server umount lustre-OST0000 complete [ 5667.296679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5667.305111] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5667.326097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5667.337162] Lustre: Skipped 1 previous similar message [ 5671.907131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5671.910795] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5673.551212] Lustre: server umount lustre-MDT0000 complete [ 5677.025348] LustreError: 184529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5677.042740] LustreError: 184529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5677.168413] LustreError: 184514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438367 with bad export cookie 18037062127959393675 [ 5677.171784] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5677.182609] LustreError: 184514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5677.520155] Lustre: server umount lustre-MDT0001 complete [ 5689.554780] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 5692.588737] Key type lgssc unregistered [ 5692.912683] LNet: 188091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5692.923631] LNetError: 188091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5692.944575] LNet: Removed LNI 192.168.203.115@tcp [ 5693.739453] Key type .llcrypt unregistered [ 5693.741691] Key type ._llcrypt unregistered [ 5716.990292] Key type ._llcrypt registered [ 5716.992677] Key type .llcrypt registered [ 5717.080427] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 5717.725740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5717.772087] alg: No test for adler32 (adler32-zlib) [ 5718.797677] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 5719.003895] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 5720.663186] Key type lgssc registered [ 5721.993202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5730.927706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5730.953176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5732.678364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5737.778640] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5743.074126] LustreError: 189510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5743.096927] LustreError: 189510:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5746.764662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5747.047656] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5750.460804] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5756.897609] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5762.379715] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5768.950298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5769.318064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5776.197254] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5777.616961] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5784.090615] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5792.146722] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 5792.228172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5792.248611] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.255452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5793.264379] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5793.267038] Lustre: Skipped 1 previous similar message [ 5793.278063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5797.204774] Lustre: lustre-OST0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 5797.753287] Lustre: server umount lustre-OST0000 complete [ 5808.759362] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff99d1984fc800.ost_server_uuid 50 [ 5810.378361] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5829.091985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5829.092152] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5829.112153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5829.126283] Lustre: Skipped 1 previous similar message [ 5831.116934] Lustre: server umount lustre-MDT0000 complete [ 5834.210533] LustreError: 189506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5834.232296] LustreError: 189506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5835.646967] LustreError: 190245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438525 with bad export cookie 4909803902441130087 [ 5835.654596] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5835.673500] LustreError: 190245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5836.170542] Lustre: server umount lustre-MDT0001 complete [ 5844.788319] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:42:12 (1787438532) [ 5851.368321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5851.819203] LustreError: 193435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5851.886422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5856.308514] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5857.248499] LustreError: 193436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.408541] LustreError: 193435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5864.480542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5864.882178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5868.815091] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5877.453543] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5882.932906] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5889.288484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5889.514438] LustreError: 195255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5889.662804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.166128] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5896.176706] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5903.666924] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5910.439719] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5910.651343] Lustre: lustre-OST0001: new disk, initializing [ 5910.655666] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5910.662382] Lustre: 196274:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5910.775740] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5916.715529] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 5920.822563] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5920.836934] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5920.929737] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5926.821200] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5930.979697] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5930.986307] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5930.995969] Lustre: Skipped 1 previous similar message [ 5931.000984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5935.784870] Lustre: server umount lustre-OST0000 complete [ 5936.624176] LustreError: 195256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5936.646072] LustreError: 195256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5941.933497] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5942.150396] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5946.830322] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5947.059632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5962.215746] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5962.219529] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5962.226323] LustreError: Skipped 1 previous similar message [ 5962.227643] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5962.241618] Lustre: Skipped 2 previous similar messages [ 5962.247537] Lustre: Skipped 3 previous similar messages [ 5967.255217] Lustre: server umount lustre-OST0001 complete [ 5972.450321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5972.450414] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5972.463648] Lustre: Skipped 1 previous similar message [ 5977.575894] LustreError: 194488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5977.619178] LustreError: 194488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 5977.694300] Lustre: server umount lustre-MDT0000 complete [ 5984.532077] LustreError: 193418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438674 with bad export cookie 4909803902441131165 [ 5984.534696] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5984.555800] LustreError: 193418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5984.927426] Lustre: server umount lustre-MDT0001 complete [ 6011.195126] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:44:59 (1787438699) [ 6018.059868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6018.480431] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6018.532500] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6018.681664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6023.831613] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6036.004734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6036.313093] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6036.406567] Lustre: 199712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6042.645494] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6051.437669] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6056.868115] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6064.236130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6064.478319] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6070.636570] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6070.933421] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6077.425814] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6083.388737] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6083.444074] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6083.706153] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6083.711501] Lustre: Skipped 2 previous similar messages [ 6089.454297] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6096.149346] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6101.474853] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6101.500805] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6101.521620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6101.526339] Lustre: Skipped 3 previous similar messages [ 6113.247133] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6113.366750] Lustre: server umount lustre-OST0000 complete [ 6117.708068] LustreError: 201565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6117.726818] LustreError: 201565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6135.836885] Lustre: server umount lustre-OST0001 complete [ 6143.458092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6143.461657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6143.474036] Lustre: Skipped 13 previous similar messages [ 6143.474702] Lustre: Skipped 4 previous similar messages [ 6145.785166] Lustre: server umount lustre-MDT0000 complete [ 6149.664106] LustreError: 200461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787438839 with bad export cookie 4909803902441132572 [ 6149.674757] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6149.683883] LustreError: 200461:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6150.183156] Lustre: server umount lustre-MDT0001 complete [ 6174.568254] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:47:42 (1787438862) [ 6190.941551] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 6201.578583] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 6249.902476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6250.139605] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6250.163172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6250.251823] Lustre: lustre-MDT0000: new disk, initializing [ 6250.331964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6250.346203] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6255.083787] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6265.150579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6265.239460] Lustre: 207815:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6265.280868] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6265.284637] Lustre: Skipped 1 previous similar message [ 6265.400669] Lustre: lustre-MDT0001: new disk, initializing [ 6265.531357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6265.539655] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6269.581225] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6279.994437] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6286.563918] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6293.685849] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6294.004522] Lustre: lustre-OST0000: new disk, initializing [ 6294.008515] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6294.013074] Lustre: 209785:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6299.783921] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6303.295468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6303.304376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6303.442202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6308.119483] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6312.165577] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6312.382337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6315.840884] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6316.069255] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6319.084489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6319.085416] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6319.092583] LustreError: Skipped 4 previous similar messages [ 6319.093157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6319.093161] Lustre: Skipped 1 previous similar message [ 6319.141042] Lustre: Skipped 1 previous similar message [ 6323.948989] Lustre: server umount lustre-OST0000 complete [ 6333.784746] Lustre: server umount lustre-MDT0000 complete [ 6334.439208] LustreError: 207828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6334.465295] LustreError: 207828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6337.679286] LustreError: 207807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439027 with bad export cookie 4909803902441134140 [ 6337.680491] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6337.690954] LustreError: 207807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6338.088775] Lustre: server umount lustre-MDT0001 complete [ 6346.756466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.083166] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6359.735531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6363.788515] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6369.977564] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6375.666072] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6383.013903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6391.989422] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6403.625900] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6409.222897] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6409.437218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6414.108882] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6414.294220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6422.397596] Lustre: server umount lustre-OST0000 complete [ 6427.901637] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6431.778039] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6463.035539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6473.716321] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6482.383392] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6482.991049] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.010804] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6483.013519] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6484.047195] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6488.035983] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6488.049994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6488.052191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6488.052200] Lustre: Skipped 5 previous similar messages [ 6488.057248] Lustre: Skipped 5 previous similar messages [ 6490.159484] Lustre: server umount lustre-OST0000 complete [ 6499.837685] Lustre: server umount lustre-MDT0000 complete [ 6503.101751] LustreError: 211982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439192 with bad export cookie 4909803902441135022 [ 6503.103966] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6503.110418] LustreError: 211982:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6503.638112] Lustre: server umount lustre-MDT0001 complete [ 6514.034881] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:53:21 (1787439201) [ 6521.765873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6522.328986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.333551] Lustre: Skipped 6 previous similar messages [ 6525.988160] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6532.799963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6537.456486] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6545.057993] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6549.921210] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6555.359429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.097395] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6568.943853] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6573.011686] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6573.269936] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6577.759587] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6577.924490] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6583.946988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.152357] Lustre: lustre-OST0001: new disk, initializing [ 6584.155749] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6584.160863] Lustre: 221341:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6590.170255] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6592.095125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6592.110473] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6592.225529] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6600.284755] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6604.643527] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6604.884416] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6609.311606] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6609.501627] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6617.310666] Lustre: server umount lustre-OST0001 complete [ 6618.093944] LustreError: 219793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6618.106111] LustreError: 219793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6622.278581] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6622.476190] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6626.663682] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6626.904159] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6655.530228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6661.098940] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6661.109296] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6661.109780] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6661.127201] Lustre: Skipped 2 previous similar messages [ 6661.832028] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6668.587705] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6670.307394] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6676.456075] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6676.468123] LustreError: Skipped 3 previous similar messages [ 6678.865527] Lustre: server umount lustre-OST0001 complete [ 6703.640569] Lustre: server umount lustre-OST0000 complete [ 6714.085657] Lustre: server umount lustre-MDT0000 complete [ 6720.436030] LustreError: 217951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439410 with bad export cookie 4909803902441135988 [ 6720.450488] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6720.465449] LustreError: 217951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6727.189912] Lustre: server umount lustre-MDT0001 complete [ 6754.406088] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:57:21 (1787439441) [ 6762.220779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6762.774163] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6762.830172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6767.843379] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6778.315378] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6778.398788] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6778.418329] Lustre: 226585:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6782.935814] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6790.315507] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6795.672633] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6801.652679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6801.837166] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6807.928138] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6815.079916] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6824.543197] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6824.639854] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6832.048963] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6839.795459] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6841.423668] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6845.638247] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6845.816553] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6850.286305] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6850.522794] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6856.022291] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff99d186457800.ost_server_uuid 50 [ 6857.523538] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff99d186457800.ost_server_uuid in IDLE state after 0 sec [ 6858.578759] Lustre: Permanently deactivating lustre-OST0001 [ 6858.601332] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6862.446715] Lustre: Permanently reactivating lustre-OST0001 [ 6866.406609] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6866.419272] Lustre: Skipped 10 previous similar messages [ 6866.426317] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6866.430645] Lustre: Skipped 15 previous similar messages [ 6868.530768] Lustre: server umount lustre-OST0001 complete [ 6886.367356] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6886.691874] Lustre: server umount lustre-OST0000 complete [ 6896.635684] Lustre: server umount lustre-MDT0000 complete [ 6900.292232] LustreError: 226576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439590 with bad export cookie 4909803902441137276 [ 6900.316234] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6900.318683] LustreError: 226576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6900.655827] Lustre: server umount lustre-MDT0001 complete [ 6930.624329] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:00:17 (1787439617) [ 6944.166540] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6944.882046] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6944.933362] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6944.943145] Lustre: Skipped 5 previous similar messages [ 6951.678496] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6961.037673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6961.269722] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6961.324592] Lustre: 233278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6964.800255] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6971.853784] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6982.028822] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6991.063622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6991.401177] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6991.434817] Lustre: Permanently deactivating lustre-OST0000 [ 6991.457172] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6998.416368] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 6999.616445] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6999.654589] LustreError: 235573:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -108 [ 7006.139207] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7013.724702] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7013.765982] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7019.958982] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7022.031289] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7030.449081] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7032.314755] Lustre: Permanently reactivating lustre-OST0000 [ 7032.331370] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7032.336467] Lustre: Skipped 2 previous similar messages [ 7039.040418] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7039.040898] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 39 seconds. I think it's dead, and I am evicting it. exp 000000001a929473, cur 1787439728 deadline 1787439719 last 1787439689 [ 7039.050498] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7039.074726] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7039.121219] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7047.970251] Lustre: server umount lustre-OST0001 complete [ 7050.722045] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7050.740158] LustreError: Skipped 7 previous similar messages [ 7059.814773] Lustre: server umount lustre-OST0000 complete [ 7071.982336] Lustre: server umount lustre-MDT0000 complete [ 7076.918287] LustreError: 236890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439766 with bad export cookie 4909803902441138921 [ 7076.932134] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7076.950560] LustreError: 236890:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7077.776543] Lustre: server umount lustre-MDT0001 complete [ 7090.734886] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7093.581506] Key type lgssc unregistered [ 7093.906819] LNet: 238177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7093.914217] LNetError: 238177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7093.925303] LNet: Removed LNI 192.168.203.115@tcp [ 7094.710127] Key type .llcrypt unregistered [ 7094.716908] Key type ._llcrypt unregistered [ 7119.812252] Key type ._llcrypt registered [ 7119.820987] Key type .llcrypt registered [ 7119.980626] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7121.056421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7121.076418] alg: No test for adler32 (adler32-zlib) [ 7122.115689] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 7122.381271] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7124.111148] Key type lgssc registered [ 7125.092738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7135.423827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7135.459963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.094322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7141.591728] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7146.521831] LustreError: 239649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7146.567492] LustreError: 239649:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7151.515090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.586362] LustreError: 239648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7151.719357] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7151.741271] Lustre: 239637:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7151.750917] Lustre: Permanently deactivating lustre-MDT0001 [ 7151.771979] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7152.023851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7152.824348] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7152.846451] LustreError: 239679:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7152.861203] LustreError: 239679:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7152.880869] LustreError: 239679:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7158.048549] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7168.033164] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7176.288906] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7183.412489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7183.657837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7190.979101] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7196.981500] LustreError: 241492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7197.003694] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7202.281872] LustreError: 241493:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7203.240920] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7206.382207] LustreError: 241492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.413064] LustreError: 241492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7206.439696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7211.126388] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7211.382283] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7216.651902] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7216.659374] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7217.627602] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7228.017315] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7232.556200] Lustre: Permanently reactivating lustre-MDT0001 [ 7232.593981] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7232.601931] Lustre: Skipped 2 previous similar messages [ 7235.316093] Lustre: lustre-MDT0001: Client 5803b9d0-f61d-4cbc-b0d4-1c24f38612c9 (at 192.168.203.15@tcp) reconnecting [ 7242.938780] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7244.768627] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7253.149393] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7253.355538] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7258.480173] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7258.763735] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7262.844859] Lustre: Permanently deactivating lustre-MDT0001 [ 7262.869468] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7262.874218] Lustre: Skipped 2 previous similar messages [ 7268.509388] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7280.109349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7280.127535] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7280.150406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.178124] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.178728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.217205] Lustre: Skipped 2 previous similar messages [ 7284.650121] Lustre: server umount lustre-MDT0000 complete [ 7288.297940] LustreError: 239643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.337933] LustreError: 239643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7288.762743] LustreError: 240388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787439978 with bad export cookie 3703953684573809319 [ 7288.770588] LustreError: 240388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7288.772849] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7289.269995] Lustre: server umount lustre-MDT0001 complete [ 7295.978804] Lustre: server umount lustre-OST0000 complete [ 7311.310180] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 19:06:39 (1787439999) [ 7326.683810] Lustre: server umount lustre-OST0001 complete [ 7335.487752] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 7348.302531] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7389.679305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7390.028208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7390.049783] Lustre: Skipped 2 previous similar messages [ 7390.106980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7390.218612] Lustre: lustre-MDT0000: new disk, initializing [ 7390.327526] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7390.341328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7395.447088] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7406.886986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7407.068756] Lustre: 247910:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7407.109196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7407.112012] Lustre: Skipped 1 previous similar message [ 7407.233724] Lustre: lustre-MDT0001: new disk, initializing [ 7407.357955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7407.409124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7407.440606] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7412.441584] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7423.566852] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7430.361956] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7437.659748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7437.950141] Lustre: lustre-OST0000: new disk, initializing [ 7437.959555] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7437.971551] Lustre: 249880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7438.045625] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7444.025316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7444.040328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7444.171370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7444.529068] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7453.898149] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7456.066528] LustreError: 247918:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7464.440041] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7464.668204] Lustre: lustre-OST0001: new disk, initializing [ 7464.670875] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7464.681292] Lustre: 250995:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7466.119081] LustreError: 247918:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7466.159116] LustreError: 247917:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7470.520458] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7470.727475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7470.737641] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7470.914156] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7476.247146] LustreError: 247917:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7480.309721] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7482.180235] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7486.432926] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7486.438465] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.452046] LustreError: Skipped 1 previous similar message [ 7486.455187] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7486.481926] Lustre: Skipped 1 previous similar message [ 7486.485882] Lustre: Skipped 3 previous similar messages [ 7491.575397] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7491.577397] Lustre: Skipped 1 previous similar message [ 7496.694235] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7496.700696] Lustre: Skipped 1 previous similar message [ 7499.231704] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7499.413340] Lustre: server umount lustre-OST0001 complete [ 7501.797866] LustreError: 249863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7501.811665] LustreError: 249863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7521.249082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7521.252619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7521.260615] LustreError: Skipped 1 previous similar message [ 7521.261703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7521.309013] Lustre: Skipped 1 previous similar message [ 7523.435776] Lustre: server umount lustre-OST0000 complete [ 7530.464243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7530.476193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7537.638157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7537.653513] Lustre: Skipped 7 previous similar messages [ 7542.239752] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7542.403448] Lustre: server umount lustre-MDT0000 complete [ 7542.764347] LustreError: 247922:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7542.783122] LustreError: 247922:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7546.202652] LustreError: 247902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440236 with bad export cookie 3703953684573812126 [ 7546.210940] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7546.223246] LustreError: 247902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7552.648324] Lustre: server umount lustre-MDT0001 complete [ 7561.283466] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7563.862201] Key type lgssc unregistered [ 7564.086651] LNet: 253033:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7564.096935] LNetError: 253033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7564.114192] LNet: Removed LNI 192.168.203.115@tcp [ 7565.007653] Key type .llcrypt unregistered [ 7565.010099] Key type ._llcrypt unregistered [ 7611.446809] Key type ._llcrypt registered [ 7611.449631] Key type .llcrypt registered [ 7611.552777] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7612.611598] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7612.640646] alg: No test for adler32 (adler32-zlib) [ 7613.784327] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 7614.248044] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7615.959653] Key type lgssc registered [ 7617.448985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7627.219992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7627.271787] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7628.624875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7628.651611] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7628.789738] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.569330] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7644.861686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7645.075818] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7645.145770] Lustre: 255243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7645.466694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7649.875912] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7659.331313] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7666.489564] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7673.966656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7674.189418] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7674.389603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7680.709046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7681.564989] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7689.467122] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7742.452516] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7742.453873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7742.472856] LustreError: Skipped 1 previous similar message [ 7742.499637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7742.505560] Lustre: Skipped 1 previous similar message [ 7742.520437] Lustre: Skipped 1 previous similar message [ 7747.561432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7747.570338] Lustre: Skipped 1 previous similar message [ 7748.324454] Lustre: server umount lustre-OST0000 complete [ 7752.788419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7766.150150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7766.451321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7772.588468] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7772.624690] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7772.631242] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7773.761783] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7782.001246] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7823.842821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7823.853454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.857674] LustreError: Skipped 1 previous similar message [ 7823.859915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7823.905022] Lustre: Skipped 1 previous similar message [ 7826.915290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7826.923623] Lustre: Skipped 2 previous similar messages [ 7828.461176] Lustre: server umount lustre-OST0000 complete [ 7835.110211] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7835.122985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7835.146349] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7837.158238] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.046396] Lustre: server umount lustre-MDT0000 complete [ 7843.950829] LustreError: 255237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440533 with bad export cookie 8303951919526729745 [ 7843.951859] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7843.963386] LustreError: 255237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7844.486762] Lustre: server umount lustre-MDT0001 complete [ 7860.310161] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 7863.180410] Key type lgssc unregistered [ 7863.445216] LNet: 260674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7863.461266] LNetError: 260674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7863.473849] LNet: Removed LNI 192.168.203.115@tcp [ 7864.094645] Key type .llcrypt unregistered [ 7864.098815] Key type ._llcrypt unregistered [ 7890.352241] Key type ._llcrypt registered [ 7890.354513] Key type .llcrypt registered [ 7890.477802] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 7891.948396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7892.031224] alg: No test for adler32 (adler32-zlib) [ 7893.102876] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 7893.302677] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 7894.967199] Key type lgssc registered [ 7896.341889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7904.802468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7904.819275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7906.349678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7910.866765] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7916.512508] LustreError: 262091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7916.537039] LustreError: 262091:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7920.717737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7921.204175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7926.249565] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7934.676186] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7940.753147] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7948.498084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7948.826438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7951.930318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7955.528568] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 7963.330483] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7990.753848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7990.768120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7990.793196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7990.822239] Lustre: Skipped 1 previous similar message [ 7995.875086] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7995.889307] Lustre: Skipped 1 previous similar message [ 7996.625785] Lustre: server umount lustre-OST0000 complete [ 8003.043134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8003.054078] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.085655] Lustre: Skipped 1 previous similar message [ 8003.102928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8006.113487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8006.119110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8006.162785] Lustre: Skipped 1 previous similar message [ 8007.848210] Lustre: server umount lustre-MDT0000 complete [ 8011.244692] LustreError: 263422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8011.284084] LustreError: 263422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8013.151980] LustreError: 262073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440703 with bad export cookie 4551577676509498078 [ 8013.166341] LustreError: 262073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8013.166823] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8013.727571] Lustre: server umount lustre-MDT0001 complete [ 8028.045783] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8031.522267] Key type lgssc unregistered [ 8031.786652] LNet: 266404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8031.801577] LNetError: 266404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8031.812895] LNet: Removed LNI 192.168.203.115@tcp [ 8032.732291] Key type .llcrypt unregistered [ 8032.734851] Key type ._llcrypt unregistered [ 8051.684588] Key type ._llcrypt registered [ 8051.687361] Key type .llcrypt registered [ 8051.798440] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8052.754776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8052.883852] alg: No test for adler32 (adler32-zlib) [ 8053.946411] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 8054.254586] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8055.984429] Key type lgssc registered [ 8057.338382] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8065.438405] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8065.458139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.934368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8070.968542] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8076.261169] LustreError: 267629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8076.288612] LustreError: 267629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8079.589730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8079.901116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8084.288063] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8091.995925] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8098.508953] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8105.504610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8105.936405] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8113.161550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8114.187668] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8123.130422] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8138.722262] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8138.734376] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8138.748707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.745372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.749672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8139.752044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.756513] Lustre: Skipped 1 previous similar message [ 8142.106952] Lustre: server umount lustre-OST0000 complete [ 8149.985898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.015497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8150.024026] Lustre: Skipped 1 previous similar message [ 8152.728052] Lustre: server umount lustre-MDT0000 complete [ 8155.124755] LustreError: 269062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.145020] LustreError: 269062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8156.702500] LustreError: 267608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440846 with bad export cookie 16649620149577181302 [ 8156.712673] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8156.715800] LustreError: 267608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8157.082569] Lustre: server umount lustre-MDT0001 complete [ 8168.523481] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8171.092185] Key type lgssc unregistered [ 8171.387348] LNet: 271308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8171.392133] LNetError: 271308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8171.431748] LNet: Removed LNI 192.168.203.115@tcp [ 8172.315813] Key type .llcrypt unregistered [ 8172.322439] Key type ._llcrypt unregistered [ 8198.251796] Key type ._llcrypt registered [ 8198.258463] Key type .llcrypt registered [ 8198.408442] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8199.458068] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8199.476551] alg: No test for adler32 (adler32-zlib) [ 8200.522587] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 8200.743623] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8202.383167] Key type lgssc registered [ 8203.477577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8210.946635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8210.972259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8212.413479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8217.081844] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8222.699185] LustreError: 272726:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8222.720493] LustreError: 272726:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8225.768029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8226.263508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8230.587200] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8238.088412] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8244.283271] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8251.316443] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8251.673694] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8257.844309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8257.920464] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8265.692313] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8288.743279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.743372] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8288.760823] LustreError: Skipped 1 previous similar message [ 8288.764327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8291.378705] Lustre: server umount lustre-OST0000 complete [ 8297.958247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8297.970309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8297.985086] Lustre: Skipped 1 previous similar message [ 8297.992566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.999467] Lustre: Skipped 1 previous similar message [ 8298.976905] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8298.998635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8299.007675] Lustre: Skipped 1 previous similar message [ 8301.284428] Lustre: server umount lustre-MDT0000 complete [ 8304.097431] LustreError: 274568:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8304.118279] LustreError: 274568:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8304.240207] LustreError: 272707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787440994 with bad export cookie 4959363485797553990 [ 8304.246642] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8304.248808] LustreError: 272707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8304.643607] Lustre: server umount lustre-MDT0001 complete [ 8315.417836] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8318.986662] Key type lgssc unregistered [ 8319.281202] LNet: 277034:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8319.297761] LNetError: 277034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8319.312006] LNet: Removed LNI 192.168.203.115@tcp [ 8320.430915] Key type .llcrypt unregistered [ 8320.435319] Key type ._llcrypt unregistered [ 8338.003029] Key type ._llcrypt registered [ 8338.006854] Key type .llcrypt registered [ 8338.086419] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8338.977478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8339.170908] alg: No test for adler32 (adler32-zlib) [ 8340.370817] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 8340.656796] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8342.359346] Key type lgssc registered [ 8343.562738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8351.122270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8351.157496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8352.661797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8356.619945] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8362.978877] LustreError: 278269:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8362.993847] LustreError: 278269:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8364.751133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8365.210804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8369.691771] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8377.981484] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8383.815633] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8391.245943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8391.488932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8397.618973] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8397.678515] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8405.116968] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8422.369102] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8422.393562] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8422.394236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8422.402602] Lustre: Skipped 1 previous similar message [ 8425.799809] Lustre: server umount lustre-OST0000 complete [ 8431.584356] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8431.594420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8431.613721] Lustre: Skipped 1 previous similar message [ 8431.624871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8432.609491] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.730626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8437.738117] Lustre: Skipped 2 previous similar messages [ 8442.850237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8442.862130] Lustre: Skipped 1 previous similar message [ 8444.383140] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8444.749361] Lustre: server umount lustre-MDT0000 complete [ 8447.979703] LustreError: 278281:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8448.013256] LustreError: 278281:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8449.074736] LustreError: 279028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441138 with bad export cookie 15042345502006306943 [ 8449.084313] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8449.092347] LustreError: 279028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8449.609801] Lustre: server umount lustre-MDT0001 complete [ 8461.572650] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8464.867238] Key type lgssc unregistered [ 8465.259150] LNet: 281975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8465.267285] LNetError: 281975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8465.283699] LNet: Removed LNI 192.168.203.115@tcp [ 8466.236315] Key type .llcrypt unregistered [ 8466.238978] Key type ._llcrypt unregistered [ 8476.914524] Key type ._llcrypt registered [ 8476.916763] Key type .llcrypt registered [ 8477.059879] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8495.528986] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8511.809740] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8512.991387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8513.089261] alg: No test for adler32 (adler32-zlib) [ 8514.282499] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 8514.487589] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8516.183163] Key type lgssc registered [ 8517.813712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8557.660492] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8557.678311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8558.929580] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8558.954399] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8559.037880] Lustre: lustre-MDT0000: new disk, initializing [ 8559.110685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8559.123391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8562.638137] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8571.557919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8571.687356] Lustre: 285531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8571.743530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8571.748160] Lustre: Skipped 1 previous similar message [ 8571.810369] Lustre: lustre-MDT0001: new disk, initializing [ 8571.903399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8571.976700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8571.999952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8577.182747] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8586.590974] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8593.211521] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8600.146415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8600.433098] Lustre: lustre-OST0000: new disk, initializing [ 8600.437521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8600.445251] Lustre: 287497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8600.582738] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8607.659432] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8608.336553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8608.341589] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8608.454154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8617.421940] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8622.967882] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8623.373482] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8628.876884] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8629.121285] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8634.361669] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8634.363156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8634.389458] LustreError: Skipped 1 previous similar message [ 8634.393251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8634.410103] Lustre: Skipped 1 previous similar message [ 8634.432725] Lustre: Skipped 1 previous similar message [ 8637.394296] Lustre: server umount lustre-OST0000 complete [ 8643.556963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8643.560990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8643.580270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.586046] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8644.611837] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8644.625415] Lustre: Skipped 1 previous similar message [ 8646.867366] Lustre: server umount lustre-MDT0000 complete [ 8650.921977] LustreError: 286323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441340 with bad export cookie 9798529772425008165 [ 8650.930162] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8650.935446] LustreError: 286323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8651.484356] Lustre: server umount lustre-MDT0001 complete [ 8661.138731] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:29:09 (1787441349) [ 8667.688861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8668.033019] LustreError: 289912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8668.055421] LustreError: 289912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8668.173074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8672.146577] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8673.248908] LustreError: 289913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8677.349101] LustreError: 289912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8679.702062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8680.050364] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8684.178854] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8690.492707] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8695.201663] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8701.007269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8701.256431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8706.505622] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8712.590353] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8722.919973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8722.924778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8722.938139] Lustre: Skipped 1 previous similar message [ 8722.950335] Lustre: Skipped 1 previous similar message [ 8726.941182] Lustre: server umount lustre-OST0000 complete [ 8731.107392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8731.113283] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8731.121871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8736.765918] Lustre: server umount lustre-MDT0000 complete [ 8738.274341] LustreError: 289907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8738.288436] LustreError: 289907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8739.547223] LustreError: 289893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441429 with bad export cookie 9798529772425009047 [ 8739.548100] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8739.555873] LustreError: 289893:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8739.763109] Lustre: server umount lustre-MDT0001 complete [ 8748.649897] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 8752.038254] Key type lgssc unregistered [ 8752.416842] LNet: 293450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8752.438131] LNetError: 293450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8752.452217] LNet: Removed LNI 192.168.203.115@tcp [ 8753.212197] Key type .llcrypt unregistered [ 8753.214636] Key type ._llcrypt unregistered [ 8787.598544] Key type ._llcrypt registered [ 8787.600899] Key type .llcrypt registered [ 8787.704467] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 8802.116845] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 8802.922510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8802.953512] alg: No test for adler32 (adler32-zlib) [ 8803.941649] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 8804.126480] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 8805.783155] Key type lgssc registered [ 8807.320255] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8853.371866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8853.409487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.666720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8854.703472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8854.806966] Lustre: lustre-MDT0000: new disk, initializing [ 8854.867407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8854.899914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8859.093458] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8868.636059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8868.745308] Lustre: 297276:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8868.801154] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8868.806382] Lustre: Skipped 1 previous similar message [ 8868.900397] Lustre: lustre-MDT0001: new disk, initializing [ 8869.016649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8869.074688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8869.098427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8873.300407] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8881.413763] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8885.463561] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8891.511554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8891.881228] Lustre: lustre-OST2710: new disk, initializing [ 8891.887984] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8891.896936] Lustre: 299245:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8892.009851] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8897.734442] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8901.194584] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8901.204625] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8901.302402] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8906.223819] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8913.060530] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8913.251581] Lustre: lustre-OST03e8: new disk, initializing [ 8913.255522] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8913.264551] Lustre: 300311:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8913.370206] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8919.195178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8919.204677] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8919.359574] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8920.934731] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 8930.807336] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8938.082354] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8938.245414] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8943.478948] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8943.700170] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8949.466346] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8949.748824] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8954.588706] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8954.702584] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8960.484615] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8960.493965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8960.500207] Lustre: Skipped 3 previous similar messages [ 8960.510310] Lustre: Skipped 3 previous similar messages [ 8964.683185] Lustre: server umount lustre-MDT0000 complete [ 8969.813064] LustreError: 297269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441659 with bad export cookie 16628782978031924785 [ 8969.833532] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8969.853631] LustreError: 297269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8970.240768] Lustre: server umount lustre-MDT0001 complete [ 8984.365319] Lustre: server umount lustre-OST2710 complete [ 8986.335269] Lustre: 295075:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787441660/real 1787441660] req@000000006f6fbb4b x1874268251034624/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787441676 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8986.385701] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8989.576987] Lustre: server umount lustre-OST03e8 complete [ 9011.124613] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [ 9022.406186] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9060.582431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9060.749692] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9060.777975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9060.855592] Lustre: lustre-MDT0000: new disk, initializing [ 9060.913615] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9060.929137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9064.907615] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9074.316247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9074.404163] Lustre: 305560:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9074.460082] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9074.467276] Lustre: Skipped 1 previous similar message [ 9074.527986] Lustre: lustre-MDT0001: new disk, initializing [ 9074.612022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9074.654107] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9074.668415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9078.725589] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9087.705418] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9093.251359] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9099.420438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9099.721724] Lustre: lustre-OST0000: new disk, initializing [ 9099.724076] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9099.731953] Lustre: 307530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9099.820208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9106.791489] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9108.018319] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9108.027313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9108.127488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9116.642092] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9121.181954] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9121.380503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9125.060796] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9125.276938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9128.937591] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9128.938029] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9128.954163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9128.968647] Lustre: Skipped 1 previous similar message [ 9134.055243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9134.072950] Lustre: Skipped 2 previous similar messages [ 9139.171569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9141.220726] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9141.560988] Lustre: server umount lustre-OST0000 complete [ 9146.342541] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9146.353798] LustreError: Skipped 1 previous similar message [ 9146.358875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9146.374494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9146.377319] Lustre: Skipped 1 previous similar message [ 9151.653545] Lustre: server umount lustre-MDT0000 complete [ 9154.532675] LustreError: 305572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.562256] LustreError: 305572:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9155.924206] LustreError: 305552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441845 with bad export cookie 16628782978031926465 [ 9155.925851] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9155.937351] LustreError: 305552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9156.465131] Lustre: server umount lustre-MDT0001 complete [ 9165.927799] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:37:33 (1787441853) [ 9167.582965] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9169.464982] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:37:37 (1787441857) [ 9173.598456] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9201.403982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9201.672860] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9201.723643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9201.943815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9206.608987] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9216.318278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9216.503233] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9216.550963] Lustre: 311389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9216.682089] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9216.689320] Lustre: Skipped 2 previous similar messages [ 9221.890673] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9243.428639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9243.669445] Lustre: Denying initial registration attempt from nid 192.168.203.115@tcp, specified as failover [ 9243.680192] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9243.690851] LustreError: 312841:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9243.712433] LustreError: 312841:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9243.718981] LustreError: 312841:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9243.773668] Lustre: server umount lustre-OST0000 complete [ 9243.776490] LustreError: 312841:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9252.832577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9252.834074] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9252.847395] Lustre: Skipped 1 previous similar message [ 9252.865479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9252.875038] Lustre: Skipped 1 previous similar message [ 9256.588877] Lustre: server umount lustre-MDT0000 complete [ 9257.952649] LustreError: 311396:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9257.982404] LustreError: 311396:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9260.008953] LustreError: 311381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787441949 with bad export cookie 16628782978031927347 [ 9260.020280] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9260.361275] Lustre: server umount lustre-MDT0001 complete [ 9271.090684] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9273.911963] Key type lgssc unregistered [ 9274.293847] LNet: 313964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9274.309177] LNetError: 313964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9275.381496] LNet: Removed LNI 192.168.203.115@tcp [ 9276.269644] Key type .llcrypt unregistered [ 9276.272029] Key type ._llcrypt unregistered [ 9290.397414] Key type ._llcrypt registered [ 9290.403164] Key type .llcrypt registered [ 9290.530556] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9292.030798] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9292.121745] alg: No test for adler32 (adler32-zlib) [ 9293.476766] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 9293.865089] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9295.527357] Key type lgssc registered [ 9296.677261] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9351.645890] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9361.348364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9361.397901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.871287] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9362.926296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9363.154563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9368.570557] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9381.956686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.083827] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9382.120396] Lustre: 316731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9382.630150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9387.921361] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9396.507163] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9403.255560] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9424.498158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9424.700725] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9424.973340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9431.234611] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9438.541745] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9442.274121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9442.277464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.306463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9442.786789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9442.809728] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9446.739706] Lustre: server umount lustre-OST0000 complete [ 9453.024865] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9453.050037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9453.056100] Lustre: Skipped 1 previous similar message [ 9457.009459] Lustre: server umount lustre-MDT0000 complete [ 9460.787962] LustreError: 316725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442150 with bad export cookie 16453336974438028035 [ 9460.790255] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9460.806286] LustreError: 316725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9461.126621] Lustre: server umount lustre-MDT0001 complete [ 9472.017626] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9475.371673] Key type lgssc unregistered [ 9475.685918] LNet: 320308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9475.699275] LNetError: 320308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9475.718378] LNet: Removed LNI 192.168.203.115@tcp [ 9476.544359] Key type .llcrypt unregistered [ 9476.548661] Key type ._llcrypt unregistered [ 9498.980298] Key type ._llcrypt registered [ 9498.984239] Key type .llcrypt registered [ 9499.110226] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9500.811263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9500.841046] alg: No test for adler32 (adler32-zlib) [ 9501.999299] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 9502.320714] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9503.975144] Key type lgssc registered [ 9505.114151] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9512.996758] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9513.026176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9514.467199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9519.286256] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9524.712938] LustreError: 321727:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9524.729572] LustreError: 321727:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9527.573759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.974341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9532.507239] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9540.754711] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9546.628648] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9556.221967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9556.608073] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9564.852440] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9573.800702] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9583.592915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9583.603791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9583.612656] Lustre: Skipped 2 previous similar messages [ 9583.620083] Lustre: Skipped 1 previous similar message [ 9586.022302] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.15@tcp (stopping) [ 9586.029883] Lustre: Skipped 1 previous similar message [ 9588.283017] Lustre: server umount lustre-MDT0000 complete [ 9588.714251] LustreError: 321722:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9588.734391] LustreError: 321722:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9591.120115] LustreError: 321724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9592.252633] LustreError: 321707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442282 with bad export cookie 4647358164859862693 [ 9592.266426] LustreError: 321707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9592.268166] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9593.825771] LustreError: 321723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9593.831420] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9593.872155] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9596.264382] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [ 9597.856218] LustreError: 321724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9597.876751] LustreError: 321724:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9598.916026] Lustre: server umount lustre-MDT0001 complete [ 9605.589325] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9613.610416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9617.387919] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x407ebba0d07bdcc7 [ 9617.417331] Lustre: MGC192.168.203.115@tcp: Connection restored to 0@lo (at 0@lo) [ 9617.828192] LustreError: 323559:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9617.926681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9618.007863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9623.032622] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9628.145881] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9632.069934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9632.504919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9637.457457] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9637.863098] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9645.198025] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9647.300513] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9653.541358] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9655.382709] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9663.463158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9663.489521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9663.493359] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9663.525557] Lustre: Skipped 2 previous similar messages [ 9666.523917] Lustre: server umount lustre-OST0000 complete [ 9673.695952] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9673.700199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9673.707279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9673.720336] Lustre: Skipped 2 previous similar messages [ 9673.734564] Lustre: Skipped 1 previous similar message [ 9676.764955] Lustre: server umount lustre-MDT0000 complete [ 9678.818183] LustreError: 325322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.851978] LustreError: 325322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9680.872589] LustreError: 321707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442370 with bad export cookie 4647358164859870407 [ 9680.880394] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9680.892866] LustreError: 321707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9681.286590] Lustre: server umount lustre-MDT0001 complete [ 9692.849170] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9695.406890] Key type lgssc unregistered [ 9695.711695] LNet: 328000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9695.723901] LNetError: 328000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9695.742306] LNet: Removed LNI 192.168.203.115@tcp [ 9696.630883] Key type .llcrypt unregistered [ 9696.635978] Key type ._llcrypt unregistered [ 9721.697523] Key type ._llcrypt registered [ 9721.699224] Key type .llcrypt registered [ 9721.768758] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [ 9722.696451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9722.768858] alg: No test for adler32 (adler32-zlib) [ 9723.952911] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [ 9724.371695] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [ 9726.050417] Key type lgssc registered [ 9727.625232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9735.540245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9735.561796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9736.866451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9741.730211] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9746.414635] LustreError: 329417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9746.445512] LustreError: 329417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9751.334781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9751.521461] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9751.768567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9756.157986] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9764.553687] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9770.925692] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9776.162700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9776.175610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9776.177760] Lustre: Skipped 1 previous similar message [ 9776.188802] Lustre: Skipped 1 previous similar message [ 9781.226223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9781.237588] Lustre: Skipped 1 previous similar message [ 9781.495803] Lustre: server umount lustre-MDT0000 complete [ 9784.765749] LustreError: 329399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442474 with bad export cookie 7604573006357158359 [ 9784.772633] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9784.780785] LustreError: 329399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9785.018966] Lustre: server umount lustre-MDT0001 complete [ 9800.352476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9800.605299] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9800.679775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9800.849383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9805.866668] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9814.542806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9814.618607] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9814.658926] Lustre: 332006:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9814.995587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9819.346971] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9827.028551] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9832.492501] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9842.221500] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9842.419506] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9842.428107] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9842.450279] LustreError: 333889:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9842.458207] LustreError: 333889:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9842.463446] LustreError: 333889:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9842.537296] Lustre: server umount lustre-OST0000 complete [ 9842.541754] LustreError: 333889:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9853.863133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9854.193659] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9854.365257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.242729] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9861.753754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9868.961236] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9876.808305] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9876.901575] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9876.995677] Lustre: lustre-OST0001: new disk, initializing [ 9877.007844] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9877.024226] Lustre: 335431:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9877.135598] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9882.201986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9882.208789] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9882.351979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9883.177732] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [ 9890.228682] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9897.962505] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9897.984613] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.987341] LustreError: Skipped 1 previous similar message [ 9897.991281] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9898.013417] Lustre: Skipped 1 previous similar message [ 9898.042647] Lustre: Skipped 1 previous similar message [ 9899.388406] Lustre: server umount lustre-OST0001 complete [ 9903.080577] LustreError: 334355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9903.093487] LustreError: 334355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9907.682049] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9907.692990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.711315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9908.220049] LustreError: 334356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9908.251910] LustreError: 334356:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9909.762974] Lustre: server umount lustre-OST0000 complete [ 9917.408726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9917.424321] LustreError: Skipped 1 previous similar message [ 9917.431607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.461530] Lustre: Skipped 1 previous similar message [ 9917.474720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9917.478025] Lustre: Skipped 2 previous similar messages [ 9920.504452] Lustre: server umount lustre-MDT0000 complete [ 9923.553333] LustreError: 332832:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9923.584570] LustreError: 332832:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9924.805986] LustreError: 331999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442614 with bad export cookie 7604573006357158912 [ 9924.813162] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9924.816386] LustreError: 331999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9925.618022] Lustre: server umount lustre-MDT0001 complete [ 9941.227206] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [ 9945.145989] Key type lgssc unregistered [ 9945.710134] LNet: 337394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9945.728745] LNetError: 337394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9945.761386] LNet: Removed LNI 192.168.203.115@tcp [ 9947.289886] Key type .llcrypt unregistered [ 9947.291482] Key type ._llcrypt unregistered [10025.537876] Key type ._llcrypt registered [10025.542315] Key type .llcrypt registered [10025.694294] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10047.395548] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10049.164350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10049.242971] alg: No test for adler32 (adler32-zlib) [10050.321922] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [10050.581867] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10052.247548] Key type lgssc registered [10053.502575] Lustre: Echo OBD driver; http://www.lustre.org/ [10092.080853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10092.109379] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10093.373327] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10093.410641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10093.509865] Lustre: lustre-MDT0000: new disk, initializing [10093.598453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10093.613488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10098.018924] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10109.025459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10109.143814] Lustre: 342332:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10109.204918] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10109.209919] Lustre: Skipped 1 previous similar message [10109.291904] Lustre: lustre-MDT0001: new disk, initializing [10109.412976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10109.449043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10109.460391] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10114.746381] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10124.794982] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10132.223193] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10140.779058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10141.189268] Lustre: lustre-OST0000: new disk, initializing [10141.195588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10141.205715] Lustre: 344303:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10141.301737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10148.139606] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10150.491542] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10150.495695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10150.574775] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10158.535917] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10163.251707] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10163.507183] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10168.240739] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10168.477273] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10171.366807] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10171.371062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10171.404095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10171.413098] Lustre: Skipped 1 previous similar message [10176.499235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10176.503404] Lustre: Skipped 1 previous similar message [10176.857070] Lustre: server umount lustre-OST0000 complete [10181.088634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10181.103331] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10181.125454] Lustre: Skipped 1 previous similar message [10181.144170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10186.727912] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10186.740548] Lustre: Skipped 2 previous similar messages [10187.234629] Lustre: server umount lustre-MDT0000 complete [10191.450733] LustreError: 342324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787442881 with bad export cookie 3429394454562583924 [10191.456238] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10191.473911] LustreError: 342324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10191.963024] Lustre: server umount lustre-MDT0001 complete [10201.534784] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:54:49 (1787442889) [10210.323671] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:54:58 (1787442898) [10220.228859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10221.214653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10226.661556] LustreError: 347059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10226.691266] LustreError: 347059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10227.311684] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10230.755284] LustreError: 347060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10235.876929] LustreError: 347059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10237.116437] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10237.624505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10242.491112] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10251.098265] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10256.995247] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10264.198835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10264.665336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10270.977250] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10278.527225] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10290.514866] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10292.138669] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10293.600341] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10295.292592] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10297.283692] Lustre: Failing over lustre-MDT0000 [10297.638737] Lustre: server umount lustre-MDT0000 complete [10298.850198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10298.852905] LustreError: 349256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10298.875829] Lustre: Skipped 3 previous similar messages [10298.877544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10298.899489] LustreError: 349256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10303.975887] LustreError: 347059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10303.993744] LustreError: 347059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10312.026848] LustreError: 347056:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10312.049859] LustreError: 347056:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10314.207417] Lustre: 340558:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787442988/real 1787442988] req@00000000df3f5aee x1874269558611200/t0(0) o400->MGC192.168.203.115@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787443004 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10314.252401] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10316.535051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.736925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10324.818692] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10327.378901] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10329.360948] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10330.092625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10330.138488] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [10330.172975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10338.145883] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10339.723659] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10343.450839] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10345.570739] Lustre: Failing over lustre-MDT0000 [10345.861114] Lustre: server umount lustre-MDT0000 complete [10347.909284] LustreError: 349256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10347.936624] LustreError: 349256:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10350.573338] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10350.573883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10365.122868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10365.243101] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10365.429893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10365.490073] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10369.353834] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10370.385284] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10370.551798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10370.560287] Lustre: Skipped 2 previous similar messages [10370.619224] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10370.714770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10380.428905] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10382.397560] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10391.011503] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10391.013504] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10391.015721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10391.034079] Lustre: Skipped 3 previous similar messages [10395.551672] Lustre: server umount lustre-MDT0000 complete [10396.150781] LustreError: 347060:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10396.188368] LustreError: 347060:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 15 previous similar messages [10399.908040] LustreError: 347040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443089 with bad export cookie 3429394454562586633 [10399.909893] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10399.919353] LustreError: 347040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10400.778634] Lustre: server umount lustre-MDT0001 complete [10416.684091] Lustre: server umount lustre-OST0000 complete [10433.704520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10434.444085] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10439.754900] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10448.858417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10453.788660] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10460.506205] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10465.968691] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10473.070187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10479.617935] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10480.174553] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10488.872996] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10494.944499] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10494.956854] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10494.958598] LustreError: Skipped 1 previous similar message [10494.964927] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10494.980378] Lustre: Skipped 2 previous similar messages [10495.024162] Lustre: Skipped 3 previous similar messages [10496.779625] Lustre: server umount lustre-OST0000 complete [10507.538173] Lustre: server umount lustre-MDT0000 complete [10510.307937] LustreError: 353428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10510.322764] LustreError: 353428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10512.242788] LustreError: 353409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443202 with bad export cookie 3429394454562587382 [10512.251783] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10512.256934] LustreError: 353409:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10512.728742] Lustre: server umount lustre-MDT0001 complete [10524.513959] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10527.692743] Key type lgssc unregistered [10528.000990] LNet: 356965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10528.012194] LNetError: 356965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10529.061769] LNet: Removed LNI 192.168.203.115@tcp [10530.037367] Key type .llcrypt unregistered [10530.039768] Key type ._llcrypt unregistered [10556.073256] Key type ._llcrypt registered [10556.076570] Key type .llcrypt registered [10556.201977] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10557.251688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10557.315852] alg: No test for adler32 (adler32-zlib) [10558.489610] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [10558.742728] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10560.415173] Key type lgssc registered [10561.537985] Lustre: Echo OBD driver; http://www.lustre.org/ [10568.923128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10568.958278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10570.544447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10575.660267] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10580.966569] LustreError: 358476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10581.001446] LustreError: 358476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10584.638572] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10585.045468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10589.525887] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10596.873522] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10602.606492] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10609.165630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10609.548543] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10615.479903] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10615.749455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10623.058360] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10635.779701] Lustre: server umount lustre-MDT0000 complete [10636.256271] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10636.268029] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10636.275906] LustreError: 358471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10636.334675] Lustre: Skipped 2 previous similar messages [10639.664024] LustreError: 358457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443329 with bad export cookie 1594452022940230726 [10639.671187] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10639.675869] LustreError: 358457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10640.015878] Lustre: server umount lustre-MDT0001 complete [10654.266473] Lustre: server umount lustre-OST0000 complete [10668.498147] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10668.905529] LustreError: 362199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10668.932063] LustreError: 362199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10668.981842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10673.227150] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10674.150652] LustreError: 362200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10678.245728] LustreError: 362199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10681.862345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10682.262203] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10686.860508] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10694.963899] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10700.797082] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10707.932644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10708.415420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10713.539684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10715.759830] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10724.146427] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10738.149613] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10738.150929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10738.154135] Lustre: Skipped 2 previous similar messages [10738.186660] Lustre: Skipped 2 previous similar messages [10743.208693] Lustre: server umount lustre-MDT0000 complete [10743.273475] LustreError: 362199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10743.305906] LustreError: 362199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10748.770064] LustreError: 364042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443438 with bad export cookie 1594452022940231839 [10748.781705] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10748.790443] LustreError: 364042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10749.305253] Lustre: server umount lustre-MDT0001 complete [10764.770739] Lustre: server umount lustre-OST0000 complete [10779.284704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.698610] LustreError: 366110:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.707770] LustreError: 366110:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10779.738369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10784.322780] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10792.721620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10797.104583] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10803.943351] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10809.632208] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10816.633790] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10817.061391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10817.067107] Lustre: Skipped 1 previous similar message [10822.968382] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [10824.243348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10829.583551] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10834.401748] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10834.409694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10834.418827] Lustre: Skipped 1 previous similar message [10834.440127] Lustre: Skipped 1 previous similar message [10836.217889] Lustre: server umount lustre-OST0000 complete [10844.129369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10844.141437] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10844.163533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10849.760959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10849.771208] Lustre: Skipped 2 previous similar messages [10854.367200] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10854.660194] Lustre: server umount lustre-MDT0000 complete [10854.883276] LustreError: 366111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.904949] LustreError: 366111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10858.392118] LustreError: 366845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443548 with bad export cookie 1594452022940232861 [10858.395239] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10858.406746] LustreError: 366845:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10858.764054] Lustre: server umount lustre-MDT0001 complete [10869.628301] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10873.050111] Key type lgssc unregistered [10873.414739] LNet: 369646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10873.417996] LNetError: 369646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10873.431148] LNet: Removed LNI 192.168.203.115@tcp [10874.073526] Key type .llcrypt unregistered [10874.075722] Key type ._llcrypt unregistered [10900.129846] Key type ._llcrypt registered [10900.133613] Key type .llcrypt registered [10900.227940] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10900.931617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10901.090068] alg: No test for adler32 (adler32-zlib) [10902.307901] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [10902.584862] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10904.280115] Key type lgssc registered [10905.328559] Lustre: Echo OBD driver; http://www.lustre.org/ [10912.020478] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10912.035381] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.043181] LustreError: 371085:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10912.058853] LustreError: 371085:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10912.066217] LustreError: 371085:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10912.071674] LustreError: 371085:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10912.079121] LustreError: 371085:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10918.346070] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10918.356388] LustreError: 371482:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10918.379637] LustreError: 371482:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10918.385780] LustreError: 371482:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10918.391068] LustreError: 371482:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10918.404923] LustreError: 371482:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10935.003521] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [10937.634478] Key type lgssc unregistered [10937.949540] LNet: 372487:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10937.963195] LNetError: 372487:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10937.992501] LNet: Removed LNI 192.168.203.115@tcp [10938.812438] Key type .llcrypt unregistered [10938.817381] Key type ._llcrypt unregistered [10956.737614] Key type ._llcrypt registered [10956.741402] Key type .llcrypt registered [10956.865157] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [10971.568988] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [10972.400103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10972.417907] alg: No test for adler32 (adler32-zlib) [10973.564110] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [10973.816655] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [10975.495165] Key type lgssc registered [10976.508326] Lustre: Echo OBD driver; http://www.lustre.org/ [11009.580634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11009.622898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11010.936860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11010.969646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11011.089583] Lustre: lustre-MDT0000: new disk, initializing [11011.149838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11011.160788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11015.646687] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11026.300089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.446946] Lustre: 375662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11026.508070] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11026.515388] Lustre: Skipped 1 previous similar message [11026.619114] Lustre: lustre-MDT0001: new disk, initializing [11026.708318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11026.746191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11026.760119] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11031.151320] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11041.391482] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11047.270855] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11054.189580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11054.502441] Lustre: lustre-OST0000: new disk, initializing [11054.506968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11054.514116] Lustre: 377629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11054.607134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11060.989747] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11063.910311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11063.926134] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11064.057034] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11070.682460] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11074.761601] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11074.968760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11079.032469] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11079.295313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11084.769933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11084.781339] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11084.806984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11084.814100] Lustre: Skipped 1 previous similar message [11089.900159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11089.906220] Lustre: Skipped 1 previous similar message [11095.010530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11095.025378] Lustre: Skipped 1 previous similar message [11096.033396] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11096.211560] Lustre: server umount lustre-OST0000 complete [11100.130764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11100.133533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11100.145100] Lustre: Skipped 2 previous similar messages [11100.157377] Lustre: Skipped 1 previous similar message [11105.253054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11105.264497] Lustre: Skipped 1 previous similar message [11106.474648] Lustre: server umount lustre-MDT0000 complete [11109.996203] LustreError: 375654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443799 with bad export cookie 8116079349123958701 [11110.007484] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11110.017117] LustreError: 375654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11110.315589] Lustre: server umount lustre-MDT0001 complete [11119.048264] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11121.027422] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:10:08 (1787443808) [11127.896781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11128.728499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11132.605317] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11133.927211] LustreError: 380092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11133.944636] LustreError: 380092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11139.042342] LustreError: 380091:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.571798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11141.961871] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11145.519080] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11152.898845] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11161.213366] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11169.676228] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11170.146432] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11176.359590] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11184.939215] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11192.466945] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11192.723823] Lustre: lustre-OST0001: new disk, initializing [11192.731153] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11192.750961] Lustre: 382927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11192.903671] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11199.494445] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11199.605696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11199.618930] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11199.778112] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11208.963684] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11215.331566] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11215.347267] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11215.360450] Lustre: Skipped 1 previous similar message [11215.382254] Lustre: Skipped 1 previous similar message [11218.906257] Lustre: server umount lustre-OST0001 complete [11220.467558] LustreError: 381913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11220.490442] LustreError: 381913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11225.579837] LustreError: 381912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11225.612177] LustreError: 381912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11230.701121] LustreError: 381914:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11230.721507] LustreError: 381914:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11237.857457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11237.864703] LustreError: Skipped 1 previous similar message [11237.868549] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11237.880756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11237.887475] Lustre: Skipped 1 previous similar message [11239.905484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11240.933703] LustreError: 382933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11240.958264] LustreError: 382933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11242.622522] Lustre: server umount lustre-OST0000 complete [11249.632315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11249.647431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11249.665197] Lustre: Skipped 1 previous similar message [11252.943698] Lustre: server umount lustre-MDT0000 complete [11257.356536] LustreError: 380071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787443947 with bad export cookie 8116079349123959583 [11257.368112] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11257.381685] LustreError: 380071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11263.918375] Lustre: server umount lustre-MDT0001 complete [11276.149467] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11279.053437] Key type lgssc unregistered [11279.340900] LNet: 384918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11279.351267] LNetError: 384918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11279.365574] LNet: Removed LNI 192.168.203.115@tcp [11280.184472] Key type .llcrypt unregistered [11280.186323] Key type ._llcrypt unregistered [11330.812959] Key type ._llcrypt registered [11330.815595] Key type .llcrypt registered [11330.915754] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11332.459161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11332.532512] alg: No test for adler32 (adler32-zlib) [11333.628904] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [11333.915489] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11335.663183] Key type lgssc registered [11336.874933] Lustre: Echo OBD driver; http://www.lustre.org/ [11344.827819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11344.847310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.156667] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11346.213132] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11346.392960] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11350.604402] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11358.183408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.316840] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11358.378142] Lustre: 387721:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11358.585982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11363.911913] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11371.108547] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11377.681248] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11384.289856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11384.293191] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11384.308602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11384.347596] Lustre: Skipped 1 previous similar message [11385.903614] Lustre: server umount lustre-MDT0000 complete [11389.970025] LustreError: 387713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444079 with bad export cookie 9791327665594673084 [11389.989570] LustreError: 387713:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11389.990024] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11390.404891] Lustre: server umount lustre-MDT0001 complete [11400.087633] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:14:48 (1787444088) [11407.365766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11407.913143] LustreError: 390231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11407.936848] LustreError: 390231:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11407.996978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11412.628825] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11413.475139] LustreError: 390232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11418.595353] LustreError: 390231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11421.425640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11421.791593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11426.383695] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11434.367271] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11439.479740] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11446.201911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11446.335866] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11446.473025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11452.637280] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11460.645564] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11468.987879] Lustre: Permanently deactivating lustre-OST0000 [11469.007041] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11477.536194] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11480.008495] LustreError: 393265:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11480.018039] LustreError: 393265:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11480.022777] LustreError: 393265:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11483.192187] LustreError: 393313:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11483.197393] LustreError: 393313:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11491.153682] Lustre: server umount lustre-OST0000 complete [11493.975201] LustreError: 393564:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11493.981175] LustreError: 393564:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11497.441622] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11497.446309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11497.466427] Lustre: Skipped 1 previous similar message [11497.485147] Lustre: Skipped 2 previous similar messages [11502.560506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11502.563404] Lustre: Skipped 1 previous similar message [11507.680617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11507.687374] Lustre: Skipped 1 previous similar message [11510.239196] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11510.389410] Lustre: server umount lustre-MDT0000 complete [11512.804660] LustreError: 392732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11512.832656] LustreError: 392732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11513.491375] LustreError: 390213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444203 with bad export cookie 9791327665594673665 [11513.499415] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11513.501729] LustreError: 390213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11524.242092] Lustre: server umount lustre-MDT0001 complete [11532.362503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11536.408652] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11538.556890] LustreError: 394725:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11543.802772] Lustre: 395064:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.115@tcp [11545.687454] Lustre: server umount MGS complete [11555.667854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11556.043708] LustreError: 395789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.064813] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11556.071724] Lustre: Skipped 1 previous similar message [11556.112692] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11560.002907] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11566.560607] LustreError: 395790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.575021] LustreError: 395790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11566.725274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11566.920654] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11570.151744] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11576.792960] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11581.779854] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11583.617837] Lustre: Permanently reactivating lustre-OST0000 [11583.637717] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11583.641966] Lustre: Skipped 2 previous similar messages [11609.567238] LustreError: 397520:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11617.071568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11617.330739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11617.340724] Lustre: Skipped 1 previous similar message [11622.938399] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11629.921857] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11634.661457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11634.669931] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11634.675314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11634.684765] Lustre: Skipped 1 previous similar message [11634.702799] Lustre: Skipped 1 previous similar message [11640.921495] Lustre: server umount lustre-OST0000 complete [11648.998813] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11649.017466] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11649.036035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11649.040684] Lustre: Skipped 2 previous similar messages [11659.231246] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11659.464733] Lustre: server umount lustre-MDT0000 complete [11660.264195] LustreError: 395789:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11660.285169] LustreError: 395789:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11663.181706] LustreError: 395770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444353 with bad export cookie 9791327665594674715 [11663.192303] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11663.193536] LustreError: 395770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11663.656230] Lustre: server umount lustre-MDT0001 complete [11673.747632] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11675.967143] Key type lgssc unregistered [11676.214518] LNet: 400204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11676.223844] LNetError: 400204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11677.289293] LNet: Removed LNI 192.168.203.115@tcp [11677.937400] Key type .llcrypt unregistered [11677.939392] Key type ._llcrypt unregistered [11693.010408] Key type ._llcrypt registered [11693.011787] Key type .llcrypt registered [11693.081506] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [11707.096153] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11708.002106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11708.174530] alg: No test for adler32 (adler32-zlib) [11709.223152] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [11709.406164] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11711.071197] Key type lgssc registered [11712.184919] Lustre: Echo OBD driver; http://www.lustre.org/ [11743.126399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11743.162841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11744.339813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11744.387294] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11744.479323] Lustre: lustre-MDT0000: new disk, initializing [11744.539451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11744.571675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11748.331654] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11757.339922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.491048] Lustre: 403361:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11757.567746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11757.577951] Lustre: Skipped 1 previous similar message [11757.648118] Lustre: lustre-MDT0001: new disk, initializing [11757.765398] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11757.831032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11757.841649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11761.896280] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11770.217322] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11775.504692] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11781.314296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11781.547757] Lustre: lustre-OST0000: new disk, initializing [11781.554565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11781.568213] Lustre: 405326:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11781.640135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11786.748502] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11787.302145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11787.318435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11787.429114] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11796.052957] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11801.155824] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11801.436735] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11805.426418] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11805.689305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11808.228621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11808.234052] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11808.250550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11808.256132] Lustre: Skipped 1 previous similar message [11811.301262] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11813.682549] Lustre: server umount lustre-OST0000 complete [11819.492889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11819.502076] LustreError: Skipped 1 previous similar message [11819.511095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11819.530847] Lustre: Skipped 1 previous similar message [11819.536176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11819.544215] Lustre: Skipped 1 previous similar message [11821.539344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11821.539621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11821.565158] Lustre: Skipped 1 previous similar message [11823.582123] Lustre: server umount lustre-MDT0000 complete [11827.682421] LustreError: 405325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444517 with bad export cookie 13080487579788000922 [11827.684031] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11827.692420] LustreError: 405325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11828.030851] Lustre: server umount lustre-MDT0001 complete [11836.626196] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11838.211785] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:22:06 (1787444526) [11843.970125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11844.438928] LustreError: 407787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11844.461417] LustreError: 407787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11844.526406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11848.946183] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11849.716513] LustreError: 407788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11854.824265] LustreError: 407787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11857.231821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11857.578887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11861.178868] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11868.547723] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11875.775657] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11883.512557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11883.937043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11890.690850] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [11899.653596] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11901.763830] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11904.332206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11910.632594] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11910.635920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11910.659066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11913.696293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11915.483217] Lustre: server umount lustre-OST0000 complete [11923.936702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11923.943465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11923.945222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11923.954961] Lustre: Skipped 2 previous similar messages [11923.958623] Lustre: Skipped 2 previous similar messages [11925.753499] Lustre: server umount lustre-MDT0000 complete [11929.067387] LustreError: 407787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11929.084205] LustreError: 407787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11929.592299] LustreError: 407769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444619 with bad export cookie 13080487579788001804 [11929.598908] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11929.610518] LustreError: 407769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11929.870840] Lustre: server umount lustre-MDT0001 complete [11939.549298] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11942.122063] Key type lgssc unregistered [11942.380669] LNet: 411514:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11942.386770] LNetError: 411514:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11942.398393] LNet: Removed LNI 192.168.203.115@tcp [11943.257612] Key type .llcrypt unregistered [11943.260184] Key type ._llcrypt unregistered [11965.994797] Key type ._llcrypt registered [11966.002869] Key type .llcrypt registered [11966.129791] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [11966.518305] Key type .llcrypt unregistered [11966.532140] Key type ._llcrypt unregistered [11985.397445] Key type ._llcrypt registered [11985.400131] Key type .llcrypt registered [11985.510403] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [11986.892076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11986.917845] alg: No test for adler32 (adler32-zlib) [11987.962559] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [11988.188648] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [11989.871218] Key type lgssc registered [11990.892697] Lustre: Echo OBD driver; http://www.lustre.org/ [11999.721289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11999.744829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12001.308247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12006.477468] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12011.495335] LustreError: 413732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12011.517471] LustreError: 413732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12015.413562] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12015.718588] LustreError: 413733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12015.733798] LustreError: 413733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12015.743393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12021.127396] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12021.225568] LustreError: 413728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12021.238143] LustreError: 413728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12026.355022] LustreError: 414478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12026.374441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12026.395699] LustreError: 414478:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12028.150441] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12031.460913] LustreError: 413728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12031.470381] LustreError: 413728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12033.846039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12034.200851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12038.520312] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12046.821062] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12046.829143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12046.841510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12048.959060] Lustre: server umount lustre-OST0000 complete [12054.496729] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12054.503676] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12054.529309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12058.672818] Lustre: server umount lustre-MDT0000 complete [12060.137278] LustreError: 413733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12060.169581] LustreError: 413733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12061.939591] LustreError: 413711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444751 with bad export cookie 14181199885816619409 [12061.944913] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12061.952736] LustreError: 413711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12071.603152] Lustre: server umount lustre-MDT0001 complete [12082.327869] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12085.369307] Key type lgssc unregistered [12085.662476] LNet: 416881:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12085.677556] LNetError: 416881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12085.696253] LNet: Removed LNI 192.168.203.115@tcp [12086.693099] Key type .llcrypt unregistered [12086.697644] Key type ._llcrypt unregistered [12109.610904] Key type ._llcrypt registered [12109.617332] Key type .llcrypt registered [12109.707192] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12110.525107] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12110.548911] alg: No test for adler32 (adler32-zlib) [12111.544391] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [12111.761857] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12113.423382] Key type lgssc registered [12114.494327] Lustre: Echo OBD driver; http://www.lustre.org/ [12121.393204] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12121.405672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12136.927356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12142.047311] LustreError: 418272:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.115@tcp: failed processing log, type 4: rc = -110 [12168.799382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12173.641118] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12179.617535] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12184.993244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12185.500215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12185.570511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12189.296372] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12190.692208] LustreError: 418297:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12190.713328] LustreError: 418297:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12195.812631] LustreError: 418298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12196.877831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12197.271983] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12201.070693] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12207.756114] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12213.950773] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12222.947316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12222.961519] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12222.980697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12223.975755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12223.981348] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12228.603235] Lustre: server umount lustre-OST0000 complete [12233.184407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12233.196268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12233.216824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12233.224246] Lustre: Skipped 1 previous similar message [12238.573326] Lustre: server umount lustre-MDT0000 complete [12239.334908] LustreError: 420420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12239.352961] LustreError: 420420:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12242.589887] LustreError: 418278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787444932 with bad export cookie 18420421439448063771 [12242.603063] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12243.029870] Lustre: server umount lustre-MDT0001 complete [12253.198565] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12255.474339] Key type lgssc unregistered [12255.744619] LNet: 421846:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12255.766656] LNetError: 421846:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12255.789848] LNet: Removed LNI 192.168.203.115@tcp [12256.528506] Key type .llcrypt unregistered [12256.533703] Key type ._llcrypt unregistered [12277.753858] Key type ._llcrypt registered [12277.755521] Key type .llcrypt registered [12277.875162] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12278.807225] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12278.892072] alg: No test for adler32 (adler32-zlib) [12279.984167] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [12280.184380] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12281.867201] Key type lgssc registered [12282.988286] Lustre: Echo OBD driver; http://www.lustre.org/ [12290.376570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12290.408537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12291.738792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12296.871276] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12301.283629] LustreError: 423244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.303210] LustreError: 423244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12304.616902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12305.031278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12309.541664] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12315.595643] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12321.058689] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12327.356794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12327.754068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12333.564477] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12336.959821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12341.121696] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12345.825164] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12345.834132] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12345.850618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12347.361412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12347.364849] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12351.496147] Lustre: server umount lustre-MDT0000 complete [12352.484726] LustreError: 425085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12352.506232] LustreError: 425085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12353.725972] LustreError: 423224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.15@tcp arrived at 1787445043 with bad export cookie 6927373868806728896 [12353.740866] LustreError: 423224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12355.543498] LustreError: 423226:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445045 with bad export cookie 6927373868806728000 [12355.556562] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12355.562380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12355.568782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12355.576745] Lustre: Skipped 1 previous similar message [12355.580719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12355.584763] Lustre: Skipped 3 previous similar messages [12357.599791] LustreError: 423245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12357.607690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12357.627864] LustreError: 423245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12360.676571] LustreError: 423241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.694945] LustreError: 423241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12361.846207] Lustre: server umount lustre-OST0000 complete [12365.795066] LustreError: 425085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12365.819653] LustreError: 425085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12368.545765] Lustre: server umount lustre-MDT0001 complete [12380.922599] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12384.500760] Key type lgssc unregistered [12384.943157] LNet: 426934:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12384.954503] LNetError: 426934:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12386.023794] LNet: Removed LNI 192.168.203.115@tcp [12387.338486] Key type .llcrypt unregistered [12387.340969] Key type ._llcrypt unregistered [12411.790571] Key type ._llcrypt registered [12411.795986] Key type .llcrypt registered [12411.904571] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12413.149486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12413.187105] alg: No test for adler32 (adler32-zlib) [12414.229575] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [12414.552276] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12416.233966] Key type lgssc registered [12417.419962] Lustre: Echo OBD driver; http://www.lustre.org/ [12425.344589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12425.366065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12426.940065] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12432.391505] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12436.486658] LustreError: 428353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12436.517640] LustreError: 428353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12441.028313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12441.386964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12446.265441] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12454.833958] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12461.037832] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12468.171884] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12468.410112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12471.495410] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12473.926237] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12480.842305] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12487.141826] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12487.166541] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12487.176524] Lustre: Skipped 1 previous similar message [12489.062403] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [12492.010711] Lustre: server umount lustre-MDT0001 complete [12492.257978] LustreError: 429083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12492.279486] LustreError: 429083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12494.157411] LustreError: 428348:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.15@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12497.385456] LustreError: 428353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12497.426095] LustreError: 428353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12502.497154] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12502.505084] LustreError: 428352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12502.515800] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12502.584468] Lustre: Skipped 1 previous similar message [12502.595083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12505.347509] Lustre: server umount lustre-OST0000 complete [12509.371091] Lustre: server umount lustre-MDT0000 complete [12523.613401] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12526.228463] Key type lgssc unregistered [12526.450913] LNet: 432042:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12526.460456] LNetError: 432042:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12526.479430] LNet: Removed LNI 192.168.203.115@tcp [12527.346188] Key type .llcrypt unregistered [12527.350073] Key type ._llcrypt unregistered [12555.845607] Key type ._llcrypt registered [12555.847668] Key type .llcrypt registered [12555.929186] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [12570.390271] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12572.111493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12572.126354] alg: No test for adler32 (adler32-zlib) [12573.265925] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [12573.704889] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12575.434283] Key type lgssc registered [12576.926574] Lustre: Echo OBD driver; http://www.lustre.org/ [12613.021438] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12626.949195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12626.988037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12628.199589] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12628.234612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12628.316821] Lustre: lustre-MDT0000: new disk, initializing [12628.424661] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12628.438371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12633.377184] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12647.037516] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12647.250390] Lustre: lustre-OST0000: new disk, initializing [12647.258568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12647.262849] Lustre: Skipped 1 previous similar message [12647.267310] Lustre: 437076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12647.342708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12652.498670] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12652.571370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12652.579435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12652.604474] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12659.961877] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12672.530806] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12672.684372] Lustre: 436058:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12672.748948] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12672.854701] Lustre: lustre-MDT0001: new disk, initializing [12672.956782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12672.999495] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12673.012570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12676.957643] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12685.607275] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12685.751223] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12689.999351] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12690.147153] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12695.920087] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.15@tcp (stopping) [12696.045643] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12696.052345] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12696.084952] Lustre: Skipped 1 previous similar message [12700.660989] Lustre: server umount lustre-MDT0001 complete [12706.276936] LustreError: 437083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12706.299867] LustreError: 437083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12711.400918] LustreError: 436070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12711.419958] LustreError: 436070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12716.516982] LustreError: 438413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12716.533406] LustreError: 438413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12719.088209] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12719.103873] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12719.133642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12719.146141] Lustre: Skipped 2 previous similar messages [12721.657038] LustreError: 436070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12721.658285] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12723.621159] Lustre: server umount lustre-OST0000 complete [12726.753411] LustreError: 438413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12733.619877] Lustre: server umount lustre-MDT0000 complete [12752.104566] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12755.157638] Key type lgssc unregistered [12755.513753] LNet: 440829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12755.527473] LNetError: 440829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12755.560421] LNet: Removed LNI 192.168.203.115@tcp [12756.360135] Key type .llcrypt unregistered [12756.363271] Key type ._llcrypt unregistered [12796.144261] Key type ._llcrypt registered [12796.148476] Key type .llcrypt registered [12796.259068] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [12796.533488] Key type .llcrypt unregistered [12796.537748] Key type ._llcrypt unregistered [12810.226349] Key type ._llcrypt registered [12810.228205] Key type .llcrypt registered [12810.330582] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [12811.217457] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12811.300421] alg: No test for adler32 (adler32-zlib) [12812.355317] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [12812.551313] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [12814.215179] Key type lgssc registered [12815.202053] Lustre: Echo OBD driver; http://www.lustre.org/ [12838.253645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12838.278345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12839.527419] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12839.597789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12839.681421] Lustre: lustre-MDT0000: new disk, initializing [12839.757917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12839.771460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12844.068280] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12854.217519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12854.381402] Lustre: 444190:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12854.467683] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12854.472162] Lustre: Skipped 1 previous similar message [12854.581150] Lustre: lustre-MDT0001: new disk, initializing [12854.727319] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12854.761192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12854.777623] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12859.219081] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12869.342127] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12875.432633] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12883.543410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12883.761888] Lustre: lustre-OST0000: new disk, initializing [12883.768428] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12883.773841] Lustre: 446158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12883.844498] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12890.199159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12890.215827] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12890.324198] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12890.375480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12899.772526] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12911.078278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12911.082706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12911.082923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12911.094097] Lustre: Skipped 2 previous similar messages [12914.316308] Lustre: server umount lustre-MDT0000 complete [12917.577094] LustreError: 444179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445607 with bad export cookie 2903063995735552429 [12917.578953] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12917.586775] LustreError: 444179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12917.900386] Lustre: server umount lustre-MDT0001 complete [12931.831669] Lustre: server umount lustre-OST0000 complete [12942.620490] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:40:30 (1787445630) [12951.709165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12952.124548] LustreError: 448185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12952.152040] LustreError: 448185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12952.223208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12956.671892] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12957.667106] LustreError: 448184:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12962.787250] LustreError: 448185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12965.440588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12965.818632] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12970.945718] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [12978.919459] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12984.744540] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13005.175697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13005.550665] Lustre: Found index 0 for lustre-OST0000, updating log [13005.832643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13011.501218] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13018.484190] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13024.239447] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13034.464662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13034.468910] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13034.477216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13034.477223] Lustre: Skipped 2 previous similar messages [13034.509164] Lustre: Skipped 1 previous similar message [13035.553133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13035.562378] Lustre: Skipped 1 previous similar message [13040.428728] Lustre: server umount lustre-OST0000 complete [13056.999114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13057.001849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13057.012825] Lustre: Skipped 1 previous similar message [13057.022422] Lustre: Skipped 1 previous similar message [13061.163849] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13061.174779] Lustre: Skipped 2 previous similar messages [13062.574554] Lustre: server umount lustre-MDT0000 complete [13066.281275] LustreError: 448166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787445756 with bad export cookie 2903063995735553822 [13066.290050] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13066.295269] LustreError: 448166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13066.717365] Lustre: server umount lustre-MDT0001 complete [13075.396213] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:42:43 (1787445763) [13076.767618] LNet: 1 local NIs in recovery (showing 1): 192.168.203.115@tcp [13692.974667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13693.405713] LustreError: 454617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13693.477865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13697.407437] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13698.537303] LustreError: 454618:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13703.651585] LustreError: 454617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13704.987912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13705.372341] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13709.612593] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13717.678756] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13723.838307] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13743.990256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13744.293992] Lustre: Found index 0 for lustre-OST0000, updating log [13744.686359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13751.196517] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [13754.655670] LNet: 1 local NIs in recovery (showing 1): 192.168.203.115@tcp [13759.067156] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13761.251765] Lustre: server umount lustre-OST0000 complete [13766.179134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13766.207401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13766.219980] Lustre: Skipped 3 previous similar messages [13767.268261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13767.277878] Lustre: Skipped 1 previous similar message [13769.316258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13769.322886] Lustre: Skipped 3 previous similar messages [13771.361233] LustreError: 456009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13771.376467] LustreError: 456009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13771.400398] Lustre: server umount lustre-MDT0000 complete [13775.293662] LustreError: 454597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787446465 with bad export cookie 2903063995735554907 [13775.304306] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13775.565156] Lustre: server umount lustre-MDT0001 complete [13806.096623] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [13813.806384] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [13848.334605] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:55:35 (1787446535) [13866.108323] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [13868.943701] Key type lgssc unregistered [13869.247799] LNet: 461226:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13869.257407] LNetError: 461226:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13997.285703] LNet: Removed LNI 192.168.203.115@tcp [13998.238531] Key type .llcrypt unregistered [13998.240663] Key type ._llcrypt unregistered [14017.401033] Key type ._llcrypt registered [14017.405063] Key type .llcrypt registered [14017.604527] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14019.244695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14019.294075] alg: No test for adler32 (adler32-zlib) [14020.354224] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [14020.546553] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [14022.239134] Key type lgssc registered [14023.799820] Lustre: Echo OBD driver; http://www.lustre.org/ [14028.536932] LNet: Added LNI 192.168.203.115@tcp100 [8/256/0/180] [14029.355582] LNet: Added LNI 192.168.203.115@tcp101 [8/256/0/180] [14030.337651] LNet: Added LNI 192.168.203.115@tcp102 [8/256/0/180] [14031.182886] LNet: Added LNI 192.168.203.115@tcp103 [8/256/0/180] [14031.912765] LNet: Added LNI 192.168.203.115@tcp104 [8/256/0/180] [14032.615617] LNet: Added LNI 192.168.203.115@tcp105 [8/256/0/180] [14033.453674] LNet: Added LNI 192.168.203.115@tcp106 [8/256/0/180] [14034.153178] LNet: Added LNI 192.168.203.115@tcp107 [8/256/0/180] [14034.962398] LNet: Added LNI 192.168.203.115@tcp108 [8/256/0/180] [14035.852089] LNet: Added LNI 192.168.203.115@tcp109 [8/256/0/180] [14036.738887] LNet: Added LNI 192.168.203.115@tcp110 [8/256/0/180] [14037.567824] LNet: Added LNI 192.168.203.115@tcp111 [8/256/0/180] [14038.455269] LNet: Added LNI 192.168.203.115@tcp112 [8/256/0/180] [14039.128673] LNet: Added LNI 192.168.203.115@tcp113 [8/256/0/180] [14040.047165] LNet: Added LNI 192.168.203.115@tcp114 [8/256/0/180] [14040.855835] LNet: Added LNI 192.168.203.115@tcp115 [8/256/0/180] [14041.729615] LNet: Added LNI 192.168.203.115@tcp116 [8/256/0/180] [14042.566500] LNet: Added LNI 192.168.203.115@tcp117 [8/256/0/180] [14043.340529] LNet: Added LNI 192.168.203.115@tcp118 [8/256/0/180] [14044.061452] LNet: Added LNI 192.168.203.115@tcp119 [8/256/0/180] [14044.891186] LNet: Added LNI 192.168.203.115@tcp120 [8/256/0/180] [14045.690378] LNet: Added LNI 192.168.203.115@tcp121 [8/256/0/180] [14046.458888] LNet: Added LNI 192.168.203.115@tcp122 [8/256/0/180] [14047.308512] LNet: Added LNI 192.168.203.115@tcp123 [8/256/0/180] [14048.171406] LNet: Added LNI 192.168.203.115@tcp124 [8/256/0/180] [14049.041882] LNet: Added LNI 192.168.203.115@tcp125 [8/256/0/180] [14049.825298] LNet: Added LNI 192.168.203.115@tcp126 [8/256/0/180] [14050.591290] LNet: Added LNI 192.168.203.115@tcp127 [8/256/0/180] [14051.565480] LNet: Added LNI 192.168.203.115@tcp128 [8/256/0/180] [14052.464200] LNet: Added LNI 192.168.203.115@tcp129 [8/256/0/180] [14053.295988] LNet: Added LNI 192.168.203.115@tcp130 [8/256/0/180] [14054.280283] LNet: Added LNI 192.168.203.115@tcp131 [8/256/0/180] [14055.278857] LNet: Added LNI 192.168.203.115@tcp132 [8/256/0/180] [14056.048283] LNet: Added LNI 192.168.203.115@tcp133 [8/256/0/180] [14056.793729] LNet: Added LNI 192.168.203.115@tcp134 [8/256/0/180] [14057.652462] LNet: Added LNI 192.168.203.115@tcp135 [8/256/0/180] [14064.026883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14064.073189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14065.234851] Lustre: 464260:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [14065.354219] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14065.374095] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14065.462256] Lustre: lustre-MDT0000: new disk, initializing [14065.568252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14065.589961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14070.197099] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14080.445654] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14080.538834] Lustre: 464278:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14080.578854] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14080.582513] Lustre: Skipped 1 previous similar message [14080.647478] Lustre: lustre-MDT0001: new disk, initializing [14080.678578] Lustre: 465077:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [14080.693387] Lustre: 465077:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14080.817793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14080.877167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14080.895794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14085.246144] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14086.214720] Lustre: 465384:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [14086.234633] Lustre: 465384:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14095.507982] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14101.697863] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14108.151825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14108.416247] Lustre: lustre-OST0000: new disk, initializing [14108.420184] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14108.427921] Lustre: 466250:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14108.510782] Lustre: 466255:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [14108.527928] Lustre: 466255:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14108.574724] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14113.577487] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14116.369482] Lustre: 466706:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.115@tcp: can't add NID 192.168.203.115@tcp130: rc = -75 [14116.384877] Lustre: 466706:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14116.434751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14116.438937] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14116.592123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14123.240457] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14137.312583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14137.322646] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14137.347826] Lustre: Skipped 1 previous similar message [14137.350068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14137.369648] Lustre: Skipped 1 previous similar message [14139.469567] Lustre: server umount lustre-OST0000 complete [14147.553825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14147.553925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14147.573649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14149.336223] Lustre: server umount lustre-MDT0000 complete [14152.990329] LustreError: 465076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787446842 with bad export cookie 7970293948806334505 [14152.992883] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14153.001597] LustreError: 465076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14153.441565] Lustre: server umount lustre-MDT0001 complete [14165.334711] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [14168.063922] Key type lgssc unregistered [14168.460147] LNet: 468166:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14168.472939] LNetError: 468166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14168.486500] LNet: Removed LNI 192.168.203.115@tcp [14169.503252] LNet: Removed LNI 192.168.203.115@tcp100 [14169.507593] LNetError: 468166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14169.513132] LNetError: 468166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14169.524130] LNet: Removed LNI 192.168.203.115@tcp101 [14170.592441] LNet: Removed LNI 192.168.203.115@tcp102 [14170.594727] LNetError: 468166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14170.600713] LNetError: 468166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14170.623175] LNet: Removed LNI 192.168.203.115@tcp103 [14171.679142] LNet: Removed LNI 192.168.203.115@tcp104 [14172.703209] LNet: Removed LNI 192.168.203.115@tcp105 [14172.717370] LNetError: 468166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14172.724913] LNetError: 468166:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14172.731857] LNet: Removed LNI 192.168.203.115@tcp106 [14173.791211] LNet: Removed LNI 192.168.203.115@tcp107 [14174.815254] LNet: Removed LNI 192.168.203.115@tcp108 [14174.817554] LNet: Removed LNI 192.168.203.115@tcp109 [14174.821721] LNet: Removed LNI 192.168.203.115@tcp110 [14174.829447] LNet: Removed LNI 192.168.203.115@tcp111 [14174.834172] LNet: Removed LNI 192.168.203.115@tcp112 [14174.842362] LNet: Removed LNI 192.168.203.115@tcp113 [14174.852188] LNet: Removed LNI 192.168.203.115@tcp114 [14174.862700] LNet: Removed LNI 192.168.203.115@tcp115 [14174.867082] LNet: Removed LNI 192.168.203.115@tcp116 [14174.871590] LNet: Removed LNI 192.168.203.115@tcp117 [14174.878992] LNet: Removed LNI 192.168.203.115@tcp118 [14174.882431] LNet: Removed LNI 192.168.203.115@tcp119 [14174.885528] LNet: Removed LNI 192.168.203.115@tcp120 [14174.888475] LNet: Removed LNI 192.168.203.115@tcp121 [14174.891202] LNet: Removed LNI 192.168.203.115@tcp122 [14174.894230] LNet: Removed LNI 192.168.203.115@tcp123 [14174.897660] LNet: Removed LNI 192.168.203.115@tcp124 [14174.899984] LNet: Removed LNI 192.168.203.115@tcp125 [14174.903354] LNet: Removed LNI 192.168.203.115@tcp126 [14174.906667] LNet: Removed LNI 192.168.203.115@tcp127 [14174.909519] LNet: Removed LNI 192.168.203.115@tcp128 [14174.912187] LNet: Removed LNI 192.168.203.115@tcp129 [14174.915156] LNet: Removed LNI 192.168.203.115@tcp130 [14174.917988] LNet: Removed LNI 192.168.203.115@tcp131 [14174.920377] LNet: Removed LNI 192.168.203.115@tcp132 [14174.922827] LNet: Removed LNI 192.168.203.115@tcp133 [14174.925417] LNet: Removed LNI 192.168.203.115@tcp134 [14174.937371] LNet: Removed LNI 192.168.203.115@tcp135 [14175.674826] Key type .llcrypt unregistered [14175.677118] Key type ._llcrypt unregistered [14196.884423] Key type ._llcrypt registered [14196.886128] Key type .llcrypt registered [14196.973769] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [14209.123739] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14210.047546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14210.092648] alg: No test for adler32 (adler32-zlib) [14211.042754] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [14211.206185] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [14212.847607] Key type lgssc registered [14213.833784] Lustre: Echo OBD driver; http://www.lustre.org/ [14241.702035] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:02:09 (1787446929) [14257.751958] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [14260.275463] Key type lgssc unregistered [14260.504198] LNet: 472243:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14260.515130] LNetError: 472243:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14260.533580] LNet: Removed LNI 192.168.203.115@tcp [14261.304225] Key type .llcrypt unregistered [14261.311839] Key type ._llcrypt unregistered [14278.595843] Key type ._llcrypt registered [14278.598988] Key type .llcrypt registered [14278.696932] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14280.030063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14280.181181] alg: No test for adler32 (adler32-zlib) [14281.174357] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [14281.377506] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [14283.039183] Key type lgssc registered [14284.078622] Lustre: Echo OBD driver; http://www.lustre.org/ [14292.925550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14292.954079] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14294.152944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14294.199381] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14294.284513] Lustre: lustre-MDT0000: new disk, initializing [14294.392320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14294.413900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14298.633239] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14308.203751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14308.310076] Lustre: 473457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14308.392817] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14308.396862] Lustre: Skipped 1 previous similar message [14308.492326] Lustre: lustre-MDT0001: new disk, initializing [14308.574446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14308.613682] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14308.624859] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14312.192765] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14320.656522] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14325.699910] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14331.157793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14331.430192] Lustre: lustre-OST0000: new disk, initializing [14331.433883] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14331.439031] Lustre: 475425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14331.528589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14336.394302] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14337.588270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14337.599322] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14337.688376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14344.390463] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14347.820355] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing lnet_if_list [14352.115638] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14352.155219] LNet: Added LNI 192.168.203.115@tcp10 [8/256/0/180] [14356.883354] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14356.924913] LNet: Added LNI 192.168.203.115@tcp11 [8/256/0/180] [14360.073221] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14360.102798] LNet: Added LNI 192.168.203.115@tcp12 [8/256/0/180] [14363.638962] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14363.667249] LNet: Added LNI 192.168.203.115@tcp13 [8/256/0/180] [14367.550963] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14367.575428] LNet: Added LNI 192.168.203.115@tcp14 [8/256/0/180] [14371.322402] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14371.345342] LNet: Added LNI 192.168.203.115@tcp15 [8/256/0/180] [14375.032203] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14375.050853] LNet: Added LNI 192.168.203.115@tcp16 [8/256/0/180] [14378.194992] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14378.224231] LNet: Added LNI 192.168.203.115@tcp17 [8/256/0/180] [14382.430512] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14382.461828] LNet: Added LNI 192.168.203.115@tcp18 [8/256/0/180] [14386.550675] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14386.579369] LNet: Added LNI 192.168.203.115@tcp19 [8/256/0/180] [14390.799886] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14390.832217] LNet: Added LNI 192.168.203.115@tcp20 [8/256/0/180] [14394.940601] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14395.000485] LNet: Added LNI 192.168.203.115@tcp21 [8/256/0/180] [14399.745260] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14399.813592] LNet: Added LNI 192.168.203.115@tcp22 [8/256/0/180] [14404.985290] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14405.017402] LNet: Added LNI 192.168.203.115@tcp23 [8/256/0/180] [14409.751825] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14409.784473] LNet: Added LNI 192.168.203.115@tcp24 [8/256/0/180] [14414.191110] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14414.222948] LNet: Added LNI 192.168.203.115@tcp25 [8/256/0/180] [14418.739480] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14418.768879] LNet: Added LNI 192.168.203.115@tcp26 [8/256/0/180] [14424.112240] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14424.135271] LNet: Added LNI 192.168.203.115@tcp27 [8/256/0/180] [14430.279389] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14430.304334] LNet: Added LNI 192.168.203.115@tcp28 [8/256/0/180] [14436.293303] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14436.344996] LNet: Added LNI 192.168.203.115@tcp29 [8/256/0/180] [14443.573067] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14443.606707] LNet: Added LNI 192.168.203.115@tcp30 [8/256/0/180] [14449.128905] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14449.160638] LNet: Added LNI 192.168.203.115@tcp31 [8/256/0/180] [14454.387593] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14454.420286] LNet: Added LNI 192.168.203.115@tcp32 [8/256/0/180] [14461.568566] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14461.609175] LNet: Added LNI 192.168.203.115@tcp33 [8/256/0/180] [14467.850738] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14467.884424] LNet: Added LNI 192.168.203.115@tcp34 [8/256/0/180] [14474.259250] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14474.293277] LNet: Added LNI 192.168.203.115@tcp35 [8/256/0/180] [14481.331856] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14481.370970] LNet: Added LNI 192.168.203.115@tcp36 [8/256/0/180] [14488.136783] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14488.155995] LNet: Added LNI 192.168.203.115@tcp37 [8/256/0/180] [14496.475605] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14496.513222] LNet: Added LNI 192.168.203.115@tcp38 [8/256/0/180] [14506.880837] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14506.909480] LNet: Added LNI 192.168.203.115@tcp39 [8/256/0/180] [14514.882958] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14514.929263] LNet: Added LNI 192.168.203.115@tcp40 [8/256/0/180] [14522.839949] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14522.897443] LNet: Added LNI 192.168.203.115@tcp41 [8/256/0/180] [14530.403804] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14530.447992] LNet: Added LNI 192.168.203.115@tcp42 [8/256/0/180] [14537.754881] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14537.781620] LNet: Added LNI 192.168.203.115@tcp43 [8/256/0/180] [14544.714948] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14544.741545] LNet: Added LNI 192.168.203.115@tcp44 [8/256/0/180] [14551.687721] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14551.736659] LNet: Added LNI 192.168.203.115@tcp45 [8/256/0/180] [14558.316759] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14558.346637] LNet: Added LNI 192.168.203.115@tcp46 [8/256/0/180] [14565.403326] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14565.441343] LNet: Added LNI 192.168.203.115@tcp47 [8/256/0/180] [14573.433844] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14573.473877] LNet: Added LNI 192.168.203.115@tcp48 [8/256/0/180] [14581.140802] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14581.171910] LNet: Added LNI 192.168.203.115@tcp49 [8/256/0/180] [14589.246924] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14589.270286] LNet: Added LNI 192.168.203.115@tcp50 [8/256/0/180] [14609.376965] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14609.381855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14609.406946] Lustre: Skipped 1 previous similar message [14609.434891] Lustre: Skipped 2 previous similar messages [14613.457987] Lustre: server umount lustre-MDT0000 complete [14617.517212] LustreError: 473447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447307 with bad export cookie 5147088679251124341 [14617.540423] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14617.541101] LustreError: 473447:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14617.986896] Lustre: server umount lustre-MDT0001 complete [14635.999161] Lustre: 472729:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787447309/real 1787447309] req@00000000c7edd485 x1874273995221376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787447325 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14636.053372] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14636.069605] Lustre: Skipped 1 previous similar message [14638.757035] Lustre: 472730:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787447312/real 1787447312] req@00000000aff3f7b4 x1874273995221504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787447328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14639.511313] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14641.122205] Lustre: 472727:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787447314/real 1787447314] req@0000000035c907a9 x1874273995221632/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787447330 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14643.171555] Lustre: 472728:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787447317/real 1787447317] req@00000000a36d8e29 x1874273995221888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787447333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14648.287470] Lustre: 472729:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787447322/real 1787447322] req@00000000d955bdcf x1874273995222016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787447338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14648.607898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14662.900509] LustreError: 475419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14662.917498] LustreError: 475419:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14662.960921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14664.993178] LustreError: 475420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14665.001306] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14666.020423] LustreError: 475419:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14666.639799] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14671.341372] LustreError: 475420:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14675.297134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14675.744963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14679.944472] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14680.912975] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14688.925270] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14694.595788] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14714.336516] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14714.344731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14714.354634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14716.899258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14716.901956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14716.905521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14716.935103] Lustre: Skipped 1 previous similar message [14722.019387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14722.033278] Lustre: Skipped 1 previous similar message [14727.135130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14727.146529] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14727.156132] Lustre: Skipped 1 previous similar message [14727.469304] Lustre: server umount lustre-OST0000 complete [14732.256811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14737.288918] Lustre: server umount lustre-MDT0000 complete [14737.382060] LustreError: 500692:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14737.407688] LustreError: 500692:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14740.891289] LustreError: 478107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447430 with bad export cookie 5147088679251126861 [14740.895058] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14740.908863] LustreError: 478107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14741.138857] Lustre: server umount lustre-MDT0001 complete [14752.050300] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [14755.223803] Key type lgssc unregistered [14755.595856] LNet: 502447:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14755.608594] LNetError: 502447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14756.640860] LNet: Removed LNI 192.168.203.115@tcp [14756.647991] LNetError: 502447:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14756.657366] LNet: Removed LNI 192.168.203.115@tcp10 [14756.659600] LNet: Removed LNI 192.168.203.115@tcp11 [14756.665157] LNet: Removed LNI 192.168.203.115@tcp12 [14756.670571] LNet: Removed LNI 192.168.203.115@tcp13 [14756.675384] LNet: Removed LNI 192.168.203.115@tcp14 [14756.679237] LNet: Removed LNI 192.168.203.115@tcp15 [14756.681465] LNet: Removed LNI 192.168.203.115@tcp16 [14756.690217] LNet: Removed LNI 192.168.203.115@tcp17 [14756.696309] LNet: Removed LNI 192.168.203.115@tcp18 [14756.705669] LNet: Removed LNI 192.168.203.115@tcp19 [14756.710308] LNet: Removed LNI 192.168.203.115@tcp20 [14756.718067] LNet: Removed LNI 192.168.203.115@tcp21 [14756.724840] LNet: Removed LNI 192.168.203.115@tcp22 [14756.731318] LNet: Removed LNI 192.168.203.115@tcp23 [14756.737121] LNet: Removed LNI 192.168.203.115@tcp24 [14756.741625] LNet: Removed LNI 192.168.203.115@tcp25 [14756.748168] LNet: Removed LNI 192.168.203.115@tcp26 [14756.757098] LNet: Removed LNI 192.168.203.115@tcp27 [14756.760697] LNet: Removed LNI 192.168.203.115@tcp28 [14756.769653] LNet: Removed LNI 192.168.203.115@tcp29 [14756.778101] LNet: Removed LNI 192.168.203.115@tcp30 [14756.783664] LNet: Removed LNI 192.168.203.115@tcp31 [14756.788085] LNet: Removed LNI 192.168.203.115@tcp32 [14756.793093] LNet: Removed LNI 192.168.203.115@tcp33 [14756.796952] LNet: Removed LNI 192.168.203.115@tcp34 [14756.805079] LNet: Removed LNI 192.168.203.115@tcp35 [14756.810768] LNet: Removed LNI 192.168.203.115@tcp36 [14756.817365] LNet: Removed LNI 192.168.203.115@tcp37 [14756.821357] LNet: Removed LNI 192.168.203.115@tcp38 [14756.824820] LNet: Removed LNI 192.168.203.115@tcp39 [14756.828725] LNet: Removed LNI 192.168.203.115@tcp40 [14756.836251] LNet: Removed LNI 192.168.203.115@tcp41 [14756.842836] LNet: Removed LNI 192.168.203.115@tcp42 [14756.848377] LNet: Removed LNI 192.168.203.115@tcp43 [14756.853446] LNet: Removed LNI 192.168.203.115@tcp44 [14756.856980] LNet: Removed LNI 192.168.203.115@tcp45 [14756.861282] LNet: Removed LNI 192.168.203.115@tcp46 [14756.864173] LNet: Removed LNI 192.168.203.115@tcp47 [14756.868154] LNet: Removed LNI 192.168.203.115@tcp48 [14756.873864] LNet: Removed LNI 192.168.203.115@tcp49 [14756.883922] LNet: Removed LNI 192.168.203.115@tcp50 [14757.627541] Key type .llcrypt unregistered [14757.629688] Key type ._llcrypt unregistered [14774.861914] Key type ._llcrypt registered [14774.864703] Key type .llcrypt registered [14774.988266] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [14792.089880] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14793.833550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14793.900428] alg: No test for adler32 (adler32-zlib) [14795.042671] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [14795.263986] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [14796.951232] Key type lgssc registered [14797.811928] Lustre: Echo OBD driver; http://www.lustre.org/ [14834.518736] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14834.556415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14835.795663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14835.816424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14835.898563] Lustre: lustre-MDT0000: new disk, initializing [14835.987662] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14836.021062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14840.224736] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14850.328236] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14850.428554] Lustre: 505646:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14850.509181] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14850.519121] Lustre: Skipped 1 previous similar message [14850.621607] Lustre: lustre-MDT0001: new disk, initializing [14850.736408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14850.766460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14850.772151] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14855.230408] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14864.058368] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14869.840686] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14877.594512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14877.771646] Lustre: *** cfs_fail_loc=90f, val=0*** [14877.820754] Lustre: lustre-OST0000: new disk, initializing [14877.824931] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14877.833995] Lustre: 507660:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14877.918791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14884.455376] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14885.964382] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14885.976577] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14886.105139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14894.419420] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14901.736445] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14901.737655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14901.749328] LustreError: Skipped 1 previous similar message [14901.765332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14906.865963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14906.879387] Lustre: Skipped 1 previous similar message [14907.565334] Lustre: server umount lustre-OST0000 complete [14911.969944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14911.988993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14911.989663] Lustre: Skipped 2 previous similar messages [14911.998604] Lustre: Skipped 1 previous similar message [14917.092396] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14917.108239] Lustre: Skipped 1 previous similar message [14918.024614] Lustre: server umount lustre-MDT0000 complete [14922.167179] LustreError: 506439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447612 with bad export cookie 9575954282980538543 [14922.171341] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14922.178590] LustreError: 506439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14922.460830] Lustre: server umount lustre-MDT0001 complete [14933.194640] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [14935.913801] Key type lgssc unregistered [14936.263601] LNet: 509569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14936.274990] LNetError: 509569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14936.326374] LNet: Removed LNI 192.168.203.115@tcp [14937.217418] Key type .llcrypt unregistered [14937.220331] Key type ._llcrypt unregistered [14956.303605] Key type ._llcrypt registered [14956.305857] Key type .llcrypt registered [14956.482659] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [14957.471743] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14957.485579] alg: No test for adler32 (adler32-zlib) [14958.735949] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [14959.088955] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [14960.792874] Key type lgssc registered [14961.999650] Lustre: Echo OBD driver; http://www.lustre.org/ [14970.038147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14970.058845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14971.571715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14976.170594] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14982.127144] LustreError: 510840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14982.157851] LustreError: 510840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14984.748101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14985.176425] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14990.749640] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [14999.196689] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15006.413254] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15014.244856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15014.616636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15021.322698] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [15030.299806] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15040.082882] loop0: detected capacity change from 0 to 204800000 [15040.321122] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15040.334373] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15040.360675] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15047.253108] loop0: detected capacity change from 0 to 204800000 [15047.429176] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15047.443170] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15047.455587] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15052.621973] loop0: detected capacity change from 0 to 204800000 [15057.639725] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15057.767441] Lustre: fs15246-OST0000: new disk, initializing [15057.771108] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15057.777296] Lustre: 514435:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15057.840908] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15065.444530] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [15076.909804] loop1: detected capacity change from 0 to 204800000 [15083.733285] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15083.976087] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15084.079973] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15084.223868] Lustre: fs15246-MDT0000: new disk, initializing [15084.442111] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15084.466265] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15084.540103] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15085.601218] Lustre: 515557:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000070728101 x1874274704998528/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787447791 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15090.136301] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [15094.710257] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15094.735172] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15094.788817] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15113.710923] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15113.725213] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15118.821911] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15119.902693] Lustre: server umount fs15246-MDT0000 complete [15123.951104] LustreError: 511789:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15129.065120] LustreError: 511789:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15129.590927] Lustre: server umount fs15246-OST0000 complete [15142.880726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15142.894682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15142.912224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15144.435418] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15147.299491] Lustre: server umount lustre-OST0000 complete [15154.145082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15154.156518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15154.167869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15154.171430] Lustre: Skipped 2 previous similar messages [15157.747850] Lustre: server umount lustre-MDT0000 complete [15159.780526] LustreError: 510839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15159.807310] LustreError: 510839:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15161.147876] LustreError: 510820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787447851 with bad export cookie 16206588148587145209 [15161.159578] LustreError: 510820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15161.162378] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15161.491478] Lustre: server umount lustre-MDT0001 complete [15172.072834] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [15174.711978] Key type lgssc unregistered [15174.991659] LNet: 518659:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15175.007350] LNetError: 518659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15175.033799] LNet: Removed LNI 192.168.203.115@tcp [15175.753194] Key type .llcrypt unregistered [15175.757510] Key type ._llcrypt unregistered [15193.910339] Key type ._llcrypt registered [15193.912882] Key type .llcrypt registered [15194.009713] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [15194.977486] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15195.067397] alg: No test for adler32 (adler32-zlib) [15196.165805] Lustre: Lustre: Build Version: 2.17.57_64_ga728441 [15196.393966] LNet: Added LNI 192.168.203.115@tcp [8/256/0/180] [15198.103171] Key type lgssc registered [15199.135930] Lustre: Echo OBD driver; http://www.lustre.org/ [15240.059868] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [15250.019618] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [15285.582211] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15285.608350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15286.880645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15286.914551] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15287.028683] Lustre: lustre-MDT0000: new disk, initializing [15287.166034] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15287.187941] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15291.478824] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [15301.401441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15301.493212] Lustre: 523128:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15301.542512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15301.547303] Lustre: Skipped 1 previous similar message [15301.552276] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15301.636675] Lustre: lustre-MDT0001: new disk, initializing [15301.717311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15301.744217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15301.763704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15305.956751] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [15315.361949] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15321.748897] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15328.863885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15329.228160] Lustre: lustre-OST0000: new disk, initializing [15329.235136] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15329.243065] Lustre: 525096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15329.352929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15335.412799] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_default_debug -1 all [15338.643113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15338.658554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15338.798762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15344.702215] Lustre: DEBUG MARKER: oleg315-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15350.361957] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15350.554906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15355.021541] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15355.235568] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15359.458872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15359.460860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15359.469513] LustreError: Skipped 1 previous similar message [15359.470750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15359.509347] Lustre: Skipped 1 previous similar message [15363.192518] Lustre: server umount lustre-OST0000 complete [15368.160867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15368.178318] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15368.196732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15368.203946] Lustre: Skipped 1 previous similar message [15369.709433] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15373.198366] Lustre: server umount lustre-MDT0000 complete [15376.621649] LustreError: 523121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787448066 with bad export cookie 13851114922434025728 [15376.623107] LustreError: MGC192.168.203.115@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15376.638211] LustreError: 523121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15376.925596] Lustre: server umount lustre-MDT0001 complete [15402.541325] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing set_hostid [15412.994612] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing load_modules_local [15444.055467] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15182 sec ========= 21:22:11 (1787448131) [15445.755371] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:22:13 (1787448133) === [15449.726685] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:22:17 (1787448137) === [15490.724789] Lustre: DEBUG MARKER: oleg315-server.virtnet: executing unload_modules_local [15493.847661] Key type lgssc unregistered [15494.163856] LNet: 531743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15494.177233] LNetError: 531743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15494.203038] LNet: Removed LNI 192.168.203.115@tcp [15495.158242] Key type .llcrypt unregistered [15495.159928] Key type ._llcrypt unregistered