[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 487407321 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.003201] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.006007] kvm-guest: setup PV IPIs [ 0.008989] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010012] pid_max: default: 32768 minimum: 301 [ 0.012042] LSM: Security Framework initializing [ 0.013000] Yama: becoming mindful. [ 0.013000] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.022160] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027773] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029005] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030120] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031118] *** VALIDATE tmpfs *** [ 0.033268] *** VALIDATE proc *** [ 0.034274] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.037128] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038240] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040038] Spectre V2 : User space: Vulnerable [ 0.041012] Speculative Store Bypass: Vulnerable [ 0.044130] debug: unmapping init [mem 0xffffffff96459000-0xffffffff96460fff] [ 0.047184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048782] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049024] ... version: 2 [ 0.050016] ... bit width: 48 [ 0.051016] ... generic registers: 4 [ 0.052017] ... value mask: 0000ffffffffffff [ 0.053016] ... max period: 00007fffffffffff [ 0.054017] ... fixed-purpose events: 3 [ 0.055015] ... event mask: 000000070000000f [ 0.056341] rcu: Hierarchical SRCU implementation. [ 0.058489] smp: Bringing up secondary CPUs ... [ 0.059625] x86: Booting SMP configuration: [ 0.060036] .... node #0, CPUs: #1 #2 #3 [ 0.064013] smp: Brought up 1 node, 4 CPUs [ 0.066023] smpboot: Max logical packages: 1 [ 0.067017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.150342] node 0 deferred pages initialised in 80ms [ 0.155021] devtmpfs: initialized [ 0.156344] x86/mm: Memory block size: 128MB [ 0.159353] gcov: version magic: 0x41383552 [ 0.162279] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166334] pinctrl core: initialized pinctrl subsystem [ 0.168167] [ 0.168894] ************************************************************* [ 0.171016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174013] ** ** [ 0.176012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.179014] ** ** [ 0.181012] ** This means that this kernel is built to expose internal ** [ 0.184016] ** IOMMU data structures, which may compromise security on ** [ 0.186011] ** your system. ** [ 0.189016] ** ** [ 0.193023] ** If you see this message and you are not debugging the ** [ 0.196016] ** kernel, report this immediately to your vendor! ** [ 0.198015] ** ** [ 0.201015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.203016] ************************************************************* [ 0.206197] NET: Registered protocol family 16 [ 0.208535] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.211083] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.214082] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.218044] cpuidle: using governor menu [ 0.222887] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.228037] PCI: Using configuration type 1 for base access [ 0.231154] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.239119] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.241083] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.243181] cryptd: max_cpu_qlen set to 1000 [ 0.247256] ACPI: Added _OSI(Module Device) [ 0.249016] ACPI: Added _OSI(Processor Device) [ 0.250009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.251014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.256378] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.262320] ACPI: Interpreter enabled [ 0.264067] ACPI: PM: (supports S0 S3 S4 S5) [ 0.265014] ACPI: Using IOAPIC for interrupt routing [ 0.267181] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.272481] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.283684] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.286071] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.289031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.294110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.300657] acpiphp: Slot [2] registered [ 0.302266] acpiphp: Slot [5] registered [ 0.304213] acpiphp: Slot [6] registered [ 0.306199] acpiphp: Slot [7] registered [ 0.307214] acpiphp: Slot [8] registered [ 0.309202] acpiphp: Slot [9] registered [ 0.311129] acpiphp: Slot [10] registered [ 0.313183] acpiphp: Slot [3] registered [ 0.314100] acpiphp: Slot [4] registered [ 0.315100] acpiphp: Slot [11] registered [ 0.317102] acpiphp: Slot [12] registered [ 0.318096] acpiphp: Slot [13] registered [ 0.319102] acpiphp: Slot [14] registered [ 0.320095] acpiphp: Slot [15] registered [ 0.322121] acpiphp: Slot [16] registered [ 0.323099] acpiphp: Slot [17] registered [ 0.324105] acpiphp: Slot [18] registered [ 0.326103] acpiphp: Slot [19] registered [ 0.327126] acpiphp: Slot [20] registered [ 0.329152] acpiphp: Slot [21] registered [ 0.331191] acpiphp: Slot [22] registered [ 0.332203] acpiphp: Slot [23] registered [ 0.334152] acpiphp: Slot [24] registered [ 0.336172] acpiphp: Slot [25] registered [ 0.337129] acpiphp: Slot [26] registered [ 0.339123] acpiphp: Slot [27] registered [ 0.340115] acpiphp: Slot [28] registered [ 0.342104] acpiphp: Slot [29] registered [ 0.343133] acpiphp: Slot [30] registered [ 0.344136] acpiphp: Slot [31] registered [ 0.346078] PCI host bridge to bus 0000:00 [ 0.347019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.349024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.351027] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.353029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.355022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.358029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.360177] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.362502] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.366575] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.376845] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.382124] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.385030] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.388035] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.391032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.395487] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.398832] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.401054] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.404838] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.411015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.424019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.429942] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.436559] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.450023] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.474022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.503022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.517900] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.526015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.536018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.554017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.567155] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.580022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.585022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.610032] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.623212] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.636018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.643017] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.662020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.674312] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.683020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.690025] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.709021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.720569] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.730110] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.741027] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.765026] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.778151] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.781487] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.784564] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.787521] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.792187] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.799100] iommu: Default domain type: Passthrough [ 0.801574] SCSI subsystem initialized [ 0.804314] ACPI: bus type USB registered [ 0.806153] usbcore: registered new interface driver usbfs [ 0.808173] usbcore: registered new interface driver hub [ 0.811112] usbcore: registered new device driver usb [ 0.813463] pps_core: LinuxPPS API ver. 1 registered [ 0.816034] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.819095] PTP clock support registered [ 0.822059] EDAC MC: Ver: 3.0.0 [ 0.824124] PCI: Using ACPI for IRQ routing [ 0.825499] NetLabel: Initializing [ 0.827016] NetLabel: domain hash size = 128 [ 0.829013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.831088] NetLabel: unlabeled traffic allowed by default [ 0.834204] vgaarb: loaded [ 0.835354] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.838021] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.848144] clocksource: Switched to clocksource kvm-clock [ 0.970063] VFS: Disk quotas dquot_6.6.0 [ 0.971974] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.974932] *** VALIDATE ramfs *** [ 0.976493] *** VALIDATE hugetlbfs *** [ 0.978365] pnp: PnP ACPI init [ 0.981087] pnp: PnP ACPI: found 6 devices [ 1.001033] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.004477] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.006799] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.009146] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.011798] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.014547] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.018095] NET: Registered protocol family 2 [ 1.020969] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.027340] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.031147] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.036689] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.040437] TCP: Hash tables configured (established 65536 bind 65536) [ 1.043826] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.047262] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.050397] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053824] NET: Registered protocol family 1 [ 1.056481] RPC: Registered named UNIX socket transport module. [ 1.058612] RPC: Registered udp transport module. [ 1.060300] RPC: Registered tcp transport module. [ 1.061874] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.064681] NET: Registered protocol family 44 [ 1.066685] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.069569] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.072217] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.075371] PCI: CLS 0 bytes, default 64 [ 1.077328] Unpacking initramfs... [ 2.499889] debug: unmapping init [mem 0xffff9c42fcc54000-0xffff9c42fffbffff] [ 2.505061] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.507593] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.510917] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.038570] Initialise system trusted keyrings [ 3.040515] Key type blacklist registered [ 3.043895] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.055119] zbud: loaded [ 3.060373] *** VALIDATE nfs *** [ 3.061872] *** VALIDATE nfs4 *** [ 3.065711] pstore: using deflate compression [ 3.069962] Platform Keyring initialized [ 3.183827] NET: Registered protocol family 38 [ 3.187073] Key type asymmetric registered [ 3.188833] Asymmetric key parser 'x509' registered [ 3.190571] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.194956] io scheduler mq-deadline registered [ 3.196778] io scheduler kyber registered [ 3.198733] io scheduler bfq registered [ 3.200507] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.203501] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.206875] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.209890] ACPI: Power Button [PWRF] [ 3.217506] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.224513] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.237885] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.245267] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.266917] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.296181] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.326329] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.332031] Non-volatile memory driver v1.3 [ 3.336325] Linux agpgart interface v0.103 [ 3.372751] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.375843] vda: detected capacity change from 0 to 76775424 [ 3.391348] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.395160] vdb: detected capacity change from 0 to 1073741824 [ 3.412739] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.415573] vdc: detected capacity change from 0 to 2621440000 [ 3.430609] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.433746] vdd: detected capacity change from 0 to 2621440000 [ 3.450513] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.454975] vde: detected capacity change from 0 to 4294967296 [ 3.473902] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.477096] vdf: detected capacity change from 0 to 4294967296 [ 3.485598] libphy: Fixed MDIO Bus: probed [ 3.494874] usbcore: registered new interface driver usbserial_generic [ 3.497943] usbserial: USB Serial support registered for generic [ 3.500469] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.505267] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.507078] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.509668] mousedev: PS/2 mouse device common for all mice [ 3.512771] rtc_cmos 00:05: RTC can wake from S4 [ 3.515529] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.520055] rtc_cmos 00:05: registered as rtc0 [ 3.522907] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.525835] intel_pstate: CPU model not supported [ 3.526664] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.532972] hid: raw HID events driver (C) Jiri Kosina [ 3.533307] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.535055] usbcore: registered new interface driver usbhid [ 3.541137] usbhid: USB HID core driver [ 3.543083] drop_monitor: Initializing network drop monitor service [ 3.545931] Initializing XFRM netlink socket [ 3.548319] NET: Registered protocol family 10 [ 3.551887] Segment Routing with IPv6 [ 3.553494] NET: Registered protocol family 17 [ 3.555967] mpls_gso: MPLS GSO support [ 3.562584] RAS: Correctable Errors collector initialized. [ 3.564842] AVX version of gcm_enc/dec engaged. [ 3.566615] AES CTR mode by8 optimization enabled [ 3.652544] sched_clock: Marking stable (3652516785, 0)->(4551110393, -898593608) [ 3.656268] registered taskstats version 1 [ 3.660588] Loading compiled-in X.509 certificates [ 3.662609] zswap: loaded using pool lzo/zbud [ 3.691563] Key type big_key registered [ 3.706325] Key type encrypted registered [ 3.708020] ima: No TPM chip found, activating TPM-bypass! [ 3.710099] ima: Allocated hash algorithm: sha1 [ 3.711723] ima: No architecture policies found [ 3.713551] evm: Initialising EVM extended attributes: [ 3.715433] evm: security.selinux [ 3.716653] evm: security.ima [ 3.717856] evm: security.capability [ 3.719292] evm: HMAC attrs: 0x1 [ 3.721750] rtc_cmos 00:05: setting system clock to 2026-09-16 23:38:30 UTC (1789601910) [ 3.728258] debug: unmapping init [mem 0xffffffff97403000-0xffffffff975fffff] [ 3.731654] debug: unmapping init [mem 0xffffffff96182000-0xffffffff96458fff] [ 3.738129] Write protecting the kernel read-only data: 28672k [ 3.741362] debug: unmapping init [mem 0xffffffff94803000-0xffffffff949fffff] [ 3.744029] debug: unmapping init [mem 0xffffffff95114000-0xffffffff951fffff] [ 3.778348] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.786902] systemd[1]: Detected virtualization kvm. [ 3.788924] systemd[1]: Detected architecture x86-64. [ 3.790967] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.819355] systemd[1]: No hostname configured. [ 3.821059] systemd[1]: Set hostname to . [ 3.823232] random: systemd: uninitialized urandom read (16 bytes read) [ 3.825590] systemd[1]: Initializing machine ID from random generator. [ 3.869465] random: ln: uninitialized urandom read (6 bytes read) [ 3.953146] random: systemd: uninitialized urandom read (16 bytes read) [ 3.955565] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.959365] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.964291] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.617554] device-mapper: uevent: version 1.0.3 [ 4.620706] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.327605] virtio_net virtio0 ens2: renamed from eth0 [ 5.352605] scsi host0: ata_piix [ 5.358838] scsi host1: ata_piix [ 5.373673] random: fast init done [ 5.409155] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.411920] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.222532] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.228711] random: crng init done [ 10.230560] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.272250] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.572656] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.856419] SELinux: Disabled at runtime. [ 12.919850] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.930218] systemd[1]: Detected virtualization kvm. [ 12.932686] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.506109] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.510336] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.516141] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.521468] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.525843] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.534087] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.544541] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ 13.720198] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 14.053189] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 14.363590] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 14.378481] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.544875] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.563150] EDAC sbridge: Ver: 1.1.2 [ 16.733566] Key type dns_resolver registered [ 17.093222] NFS: Registering the id_resolver key type [ 17.097149] Key type id_resolver registered [ 17.099187] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg443-server login: [ 64.626768] libcfs: loading out-of-tree module taints kernel. [ 64.774312] Key type ._llcrypt registered [ 64.778994] Key type .llcrypt registered [ 64.962479] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 85.170052] hrtimer: interrupt took 11044106 ns [ 88.085737] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 90.230177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 90.261698] alg: No test for adler32 (adler32-zlib) [ 91.940190] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 93.021085] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 94.767430] Key type lgssc registered [ 96.888236] Lustre: Echo OBD driver; http://www.lustre.org/ [ 117.717875] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 165.579138] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 180.011893] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 180.059881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 181.317863] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 181.351629] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 181.506295] Lustre: lustre-MDT0000: new disk, initializing [ 181.615067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 181.647566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 186.996355] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 202.492079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 202.669973] Lustre: 6520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 202.739842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 202.751918] Lustre: Skipped 1 previous similar message [ 202.929429] Lustre: lustre-MDT0001: new disk, initializing [ 203.027780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.072568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 203.100828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.941881] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.295521] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 226.135719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 226.506328] Lustre: lustre-OST0000: new disk, initializing [ 226.509983] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 226.517746] Lustre: 8458:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 226.606107] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 230.094846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 230.108695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 230.295273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.673388] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 253.211315] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 253.410991] Lustre: lustre-OST0001: new disk, initializing [ 253.417853] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 253.426233] Lustre: 9534:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 253.558702] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 260.020629] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 262.154698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 262.175615] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 262.213628] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 272.729851] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 279.784298] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 287.610853] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing check_logdir /tmp/testlogs/ [ 292.965742] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing yml_node [ 297.741843] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 301.005159] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 304.345723] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 306.613914] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 16 19:43:30 EDT 2026 [ 327.734235] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 329.578879] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 346.592223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 346.603564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 346.620608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.158281] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 349.159642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 349.199373] Lustre: Skipped 3 previous similar messages [ 352.091343] Lustre: server umount lustre-MDT0000 complete [ 355.580515] LustreError: 8456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602262 with bad export cookie 12221498303505192856 [ 355.589061] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 355.591617] LustreError: 8456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 359.392086] LustreError: 9542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 359.392374] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.414664] LustreError: 9542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 359.448738] Lustre: Skipped 2 previous similar messages [ 359.455133] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 359.459929] Lustre: Skipped 1 previous similar message [ 360.417882] LustreError: 6527:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 360.444050] LustreError: 6527:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 361.982624] Lustre: server umount lustre-MDT0001 complete [ 365.716494] Lustre: server umount lustre-OST0000 complete [ 369.428684] Lustre: server umount lustre-OST0001 complete [ 376.945674] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 387.826471] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 430.653274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 430.889604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 430.926195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 430.997815] Lustre: lustre-MDT0000: new disk, initializing [ 431.088864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 431.107644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 436.329663] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 448.594845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 448.760122] Lustre: 15088:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 448.897344] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 448.903778] Lustre: Skipped 1 previous similar message [ 449.001222] Lustre: lustre-MDT0001: new disk, initializing [ 449.113142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 449.165980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 449.180650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 454.425355] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 466.858468] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 473.528492] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 482.085610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 482.483979] Lustre: lustre-OST0000: new disk, initializing [ 482.490488] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 482.498933] Lustre: 17058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 482.607477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 484.515117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 484.527930] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 484.636821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 489.606675] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 499.305654] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 503.827383] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 504.028208] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 508.516168] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 508.883905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 515.562540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 515.565945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 515.577860] LustreError: Skipped 1 previous similar message [ 515.579098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 515.579103] Lustre: Skipped 2 previous similar messages [ 515.610638] Lustre: Skipped 2 previous similar messages [ 520.684770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.688640] Lustre: Skipped 2 previous similar messages [ 525.279158] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 525.421987] Lustre: server umount lustre-OST0000 complete [ 530.912538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 530.924956] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.936470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.968285] Lustre: Skipped 1 previous similar message [ 536.033737] LustreError: 15102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.067569] LustreError: 15102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 537.527504] Lustre: server umount lustre-MDT0000 complete [ 541.158907] LustreError: 15957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 541.190833] LustreError: 15957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 541.874889] LustreError: 15894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602448 with bad export cookie 12221498303505194389 [ 541.894790] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 541.895796] LustreError: 15894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 542.550871] Lustre: server umount lustre-MDT0001 complete [ 547.258909] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 19:47:31 (1789602451) [ 549.982748] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 551.858322] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 19:47:36 (1789602456) [ 558.358904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.863321] LustreError: 19474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 558.966160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 563.127483] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 568.298626] LustreError: 19474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.329526] LustreError: 19474:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 572.379925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.353326] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 584.650544] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 591.006459] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 598.807599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 606.397678] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 616.118225] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 628.193375] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.196410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.208060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 628.221189] Lustre: Skipped 1 previous similar message [ 632.245429] Lustre: server umount lustre-OST0000 complete [ 650.212435] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 650.643928] Lustre: server umount lustre-MDT0000 complete [ 653.799732] LustreError: 21224:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.824302] LustreError: 21224:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 654.395243] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602561 with bad export cookie 12221498303505195271 [ 654.403443] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 654.418251] LustreError: 19454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 654.873137] Lustre: server umount lustre-MDT0001 complete [ 665.773671] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 668.817725] Key type lgssc unregistered [ 669.196843] LNet: 23009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 669.209420] LNetError: 23009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 669.234322] LNet: Removed LNI 192.168.204.143@tcp [ 670.267245] Key type .llcrypt unregistered [ 670.273660] Key type ._llcrypt unregistered [ 691.205469] Key type ._llcrypt registered [ 691.212986] Key type .llcrypt registered [ 691.321795] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 692.299419] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 692.324434] alg: No test for adler32 (adler32-zlib) [ 693.330763] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 693.570851] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 695.343519] Key type lgssc registered [ 696.285440] Lustre: Echo OBD driver; http://www.lustre.org/ [ 706.016029] loop: module loaded [ 706.146446] loop0: detected capacity change from 0 to 204800000 [ 706.617965] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 706.634564] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 707.200560] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.496873] loop0: detected capacity change from 0 to 204800000 [ 715.824516] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.852415] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.893386] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 723.119534] loop0: detected capacity change from 0 to 204800000 [ 729.795592] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 729.817190] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.105508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 731.184976] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 731.285415] Lustre: lustre-MDT0000: new disk, initializing [ 731.393357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 731.424105] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 737.739771] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 746.928665] loop1: detected capacity change from 0 to 204800000 [ 754.702544] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 755.300867] Lustre: lustre-OST07c6: new disk, initializing [ 755.308318] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 755.313255] Lustre: Skipped 1 previous similar message [ 755.317985] Lustre: 26193:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 755.422504] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 761.978690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 761.989070] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 762.018844] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 762.719486] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 767.818833] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 767.830191] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 767.837064] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 767.847088] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 767.858806] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 767.866160] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 785.275238] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 785.291906] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 785.306639] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 785.323693] Lustre: 25179:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 785.346569] Lustre: 25179:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 785.355695] Lustre: 25179:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.746514] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 790.765216] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 790.773435] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 790.793413] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.813665] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 790.826624] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.839811] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 790.848533] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.858421] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 790.866118] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 790.878797] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 790.890715] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.233364] Lustre: 26858:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 794.249721] Lustre: 26858:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.256089] Lustre: 26858:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 794.264309] Lustre: 26858:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.279366] Lustre: 26858:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.304440] Lustre: 26858:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.312091] Lustre: 26858:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 794.326234] Lustre: 26858:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.335424] Lustre: 26858:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 794.352502] Lustre: 26858:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.364818] Lustre: 26858:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.378460] Lustre: 26858:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.320898] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 799.342534] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 799.357086] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 799.373376] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.387737] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 799.402945] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.412635] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 799.432344] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.439855] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 799.447269] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 799.451032] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 799.454858] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 807.637197] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 807.650174] Lustre: 25180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 807.666758] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 807.677871] Lustre: 25180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.689338] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 807.701886] Lustre: 25180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.713590] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 807.724945] Lustre: 25180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.739423] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 807.751125] Lustre: 25180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 807.759402] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 807.767785] Lustre: 25180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 823.653959] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 823.665574] Lustre: 25179:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 823.676387] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 823.687061] Lustre: 25179:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 823.696912] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 823.706258] Lustre: 25179:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 859.108878] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 859.118276] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 859.130683] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 864.239224] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 865.291686] Lustre: server umount lustre-OST07c6 complete [ 875.503869] Lustre: server umount lustre-MDT0000 complete [ 896.352969] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 900.247853] Key type lgssc unregistered [ 900.651880] LNet: 28747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 900.663179] LNetError: 28747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 900.687712] LNet: Removed LNI 192.168.204.143@tcp [ 901.851890] Key type .llcrypt unregistered [ 901.855108] Key type ._llcrypt unregistered [ 925.357463] Key type ._llcrypt registered [ 925.359810] Key type .llcrypt registered [ 925.442955] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 926.478358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 926.493958] alg: No test for adler32 (adler32-zlib) [ 927.612779] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 928.021519] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 929.818470] Key type lgssc registered [ 931.559655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 940.601868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 940.621537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.152212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.617686] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 951.295415] LustreError: 30020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 951.313623] LustreError: 30020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 955.916802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 956.415553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 962.198899] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 969.772447] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 975.220908] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 981.955185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 982.332459] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 986.420562] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 988.430881] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 995.745469] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1016.317875] Lustre: server umount lustre-OST0000 complete [ 1022.433538] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1022.440203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1022.454365] Lustre: Skipped 1 previous similar message [ 1022.467262] Lustre: Skipped 1 previous similar message [ 1027.553972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1027.562392] Lustre: Skipped 1 previous similar message [ 1030.773732] Lustre: server umount lustre-MDT0000 complete [ 1032.674853] LustreError: 30020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.701370] LustreError: 30020:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1033.696111] Lustre: 29328:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789602924/real 1789602924] req@000000004d24c6a0 x1876534581295744/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789602940 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1033.698065] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1033.733395] Lustre: 29328:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1033.750589] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1033.777115] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1033.853414] LustreError: 30752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789602940 with bad export cookie 8953540526998142550 [ 1033.857879] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1033.864412] LustreError: 30752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1034.227130] Lustre: server umount lustre-MDT0001 complete [ 1044.664676] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1047.563222] Key type lgssc unregistered [ 1047.892886] LNet: 33753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1047.913832] LNetError: 33753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1047.925646] LNet: Removed LNI 192.168.204.143@tcp [ 1048.908423] Key type .llcrypt unregistered [ 1048.911087] Key type ._llcrypt unregistered [ 1067.602992] Key type ._llcrypt registered [ 1067.607024] Key type .llcrypt registered [ 1067.770641] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1069.055688] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1069.133033] alg: No test for adler32 (adler32-zlib) [ 1070.172919] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1070.561085] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1072.271284] Key type lgssc registered [ 1073.402696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1081.027607] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1081.071039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1082.546736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1087.188934] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1093.093133] LustreError: 35019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1093.126624] LustreError: 35019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1097.051764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1097.474399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1102.080095] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1108.831742] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1114.245488] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1120.194482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1120.474664] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1126.140871] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1127.676448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1133.318301] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1148.389655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1148.392533] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.399374] LustreError: Skipped 1 previous similar message [ 1148.415448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1148.428472] Lustre: Skipped 1 previous similar message [ 1148.447634] Lustre: Skipped 1 previous similar message [ 1150.153221] Lustre: server umount lustre-OST0000 complete [ 1158.624814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.646196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1158.654524] Lustre: Skipped 1 previous similar message [ 1160.786243] Lustre: server umount lustre-MDT0000 complete [ 1163.744273] LustreError: 35018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1163.776668] LustreError: 35018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1164.830448] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603071 with bad export cookie 8078606128457075613 [ 1164.839682] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1164.855945] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1165.171261] Lustre: server umount lustre-MDT0001 complete [ 1178.859607] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1182.264410] Key type lgssc unregistered [ 1182.695241] LNet: 38549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1182.701823] LNetError: 38549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1182.747776] LNet: Removed LNI 192.168.204.143@tcp [ 1183.677355] Key type .llcrypt unregistered [ 1183.680077] Key type ._llcrypt unregistered [ 1208.876404] Key type ._llcrypt registered [ 1208.880050] Key type .llcrypt registered [ 1209.056762] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1210.460305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1210.634332] alg: No test for adler32 (adler32-zlib) [ 1211.691921] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1211.922240] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1213.631769] Key type lgssc registered [ 1214.900959] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1223.341228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1223.358820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1224.849286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1230.409826] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1234.410419] LustreError: 39973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1234.432496] LustreError: 39973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1239.528183] LustreError: 39974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1239.851638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1240.301276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1245.272713] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1253.302961] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1259.866852] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1266.934669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1267.347050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.378493] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1273.832342] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1280.689834] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1290.217526] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1290.231054] Lustre: Skipped 1 previous similar message [ 1290.236456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1290.244666] Lustre: Skipped 1 previous similar message [ 1293.636169] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 1293.642099] Lustre: Skipped 1 previous similar message [ 1295.337538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1295.358288] Lustre: Skipped 2 previous similar messages [ 1296.460119] Lustre: server umount lustre-MDT0000 complete [ 1298.739198] LustreError: 39970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.004126] LustreError: 39952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603206 with bad export cookie 2049274126448045221 [ 1300.016074] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1300.460647] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1300.482451] Lustre: Skipped 1 previous similar message [ 1300.488482] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1303.860627] LustreError: 39968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1303.873256] LustreError: 39968:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1305.573568] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1305.583612] Lustre: Skipped 1 previous similar message [ 1306.417968] Lustre: server umount lustre-MDT0001 complete [ 1327.846782] Lustre: server umount lustre-OST0000 complete [ 1342.839968] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1345.375600] Key type lgssc unregistered [ 1345.696713] LNet: 43831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1345.725443] LNetError: 43831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1345.747904] LNet: Removed LNI 192.168.204.143@tcp [ 1346.731564] Key type .llcrypt unregistered [ 1346.733835] Key type ._llcrypt unregistered [ 1373.929086] Key type ._llcrypt registered [ 1373.931331] Key type .llcrypt registered [ 1374.051543] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1375.284514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1375.324281] alg: No test for adler32 (adler32-zlib) [ 1376.470777] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1376.772956] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1378.471255] Key type lgssc registered [ 1379.718490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1387.881418] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1387.903448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1389.394291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1394.176621] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1399.782521] LustreError: 45257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1399.807710] LustreError: 45257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1403.911841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1404.435921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1410.030347] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1418.511514] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1424.978728] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1432.472822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1432.980358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1435.089683] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1440.078394] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1447.404493] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1454.567197] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1454.577686] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1454.578434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1454.585566] Lustre: Skipped 1 previous similar message [ 1455.460644] Lustre: lustre-OST0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 1455.474643] Lustre: Skipped 1 previous similar message [ 1459.692491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1459.697949] Lustre: Skipped 1 previous similar message [ 1464.803797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1464.812673] Lustre: Skipped 2 previous similar messages [ 1468.383865] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1468.510818] Lustre: server umount lustre-OST0000 complete [ 1486.308345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1486.321234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1486.350319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1486.358579] Lustre: Skipped 1 previous similar message [ 1490.402190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.644443] Lustre: server umount lustre-MDT0000 complete [ 1494.565674] LustreError: 45251:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1494.582691] LustreError: 45251:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1497.040892] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603403 with bad export cookie 13341957982660915549 [ 1497.052784] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1497.056672] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1503.438978] Lustre: server umount lustre-MDT0001 complete [ 1512.332745] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1515.020695] Key type lgssc unregistered [ 1515.294117] LNet: 48963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1515.308127] LNetError: 48963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1515.326846] LNet: Removed LNI 192.168.204.143@tcp [ 1516.182453] Key type .llcrypt unregistered [ 1516.186518] Key type ._llcrypt unregistered [ 1538.526211] Key type ._llcrypt registered [ 1538.530161] Key type .llcrypt registered [ 1538.667861] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1539.572422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1539.777251] alg: No test for adler32 (adler32-zlib) [ 1540.996232] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1541.389949] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1543.120595] Key type lgssc registered [ 1544.396351] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1553.652072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1553.676518] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1555.227941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1560.346446] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1564.645675] LustreError: 50379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1564.677727] LustreError: 50379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1568.911305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1569.342667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1574.133732] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1582.547549] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1588.258784] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1595.243479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1595.619985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1599.735566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1601.380254] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1607.533219] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1616.979710] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1620.658893] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1633.297879] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-1620h05m37s [ 1635.384560] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1640.801714] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 1640.931088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.953982] Lustre: Skipped 1 previous similar message [ 1641.843680] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 1641.850734] Lustre: Skipped 3 previous similar messages [ 1643.890160] Lustre: server umount lustre-MDT0000 complete [ 1647.733138] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1647.933224] LustreError: 50375:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.070877] LustreError: 50373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1654.094904] LustreError: 50373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1654.713032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1654.893218] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1655.262225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1655.325602] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1656.315643] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1660.898113] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1665.865191] Lustre: DEBUG MARKER: Wait for df (36702) ... [ 1667.769917] Lustre: DEBUG MARKER: done [ 1675.747831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1675.761556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1675.784596] Lustre: Skipped 1 previous similar message [ 1675.799570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1675.805892] Lustre: Skipped 3 previous similar messages [ 1676.776375] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1678.654267] Lustre: server umount lustre-OST0000 complete [ 1687.009115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1687.035113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1687.053780] Lustre: Skipped 3 previous similar messages [ 1688.705615] Lustre: server umount lustre-MDT0000 complete [ 1692.987283] LustreError: 50360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603599 with bad export cookie 16857235273473594407 [ 1693.009248] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1693.014465] LustreError: 50360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1693.970892] Lustre: server umount lustre-MDT0001 complete [ 1705.927841] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1709.407152] Key type lgssc unregistered [ 1709.676805] LNet: 55305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1709.682321] LNetError: 55305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1710.756904] LNet: Removed LNI 192.168.204.143@tcp [ 1711.442619] Key type .llcrypt unregistered [ 1711.444323] Key type ._llcrypt unregistered [ 1746.738644] Key type ._llcrypt registered [ 1746.745099] Key type .llcrypt registered [ 1746.873937] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1747.877814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1747.906539] alg: No test for adler32 (adler32-zlib) [ 1748.993245] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1749.260281] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1751.039324] Key type lgssc registered [ 1752.088555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1758.363952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1758.411572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1759.649102] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1759.662179] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1759.790493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1763.655129] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1770.590919] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1770.665382] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1770.686399] Lustre: 57489:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1770.930074] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1775.527783] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1783.305056] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1789.565466] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1797.342867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1797.556379] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1797.854697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.235270] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1806.492258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1811.269626] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1818.920946] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-1620h08m43s [ 1820.356379] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1828.361331] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1831.164243] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1831.223116] LustreError: 59645:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1851.753168] Lustre: lustre-MDT0000: Client 3b412377-0e4d-47ec-b82c-ead14c140b03 (at 192.168.204.43@tcp) reconnecting [ 1871.247096] LustreError: 59645:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1873.383371] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1875.680417] Lustre: DEBUG MARKER: done [ 1883.104625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1883.110318] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.125760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1883.618397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1883.620795] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1883.637953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.643222] Lustre: server umount lustre-OST0000 complete [ 1893.856189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1893.858083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1893.873138] Lustre: Skipped 1 previous similar message [ 1893.893165] Lustre: Skipped 1 previous similar message [ 1895.802528] Lustre: server umount lustre-MDT0000 complete [ 1898.735629] LustreError: 59361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603805 with bad export cookie 3177745840611161568 [ 1898.739064] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.749696] LustreError: 59361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1899.054324] Lustre: server umount lustre-MDT0001 complete [ 1909.327950] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 1911.701469] Key type lgssc unregistered [ 1911.949661] LNet: 61816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1911.956338] LNetError: 61816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1911.977519] LNet: Removed LNI 192.168.204.143@tcp [ 1912.760362] Key type .llcrypt unregistered [ 1912.762840] Key type ._llcrypt unregistered [ 1954.239086] Key type ._llcrypt registered [ 1954.242433] Key type .llcrypt registered [ 1954.325892] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 1955.182560] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1955.415721] alg: No test for adler32 (adler32-zlib) [ 1956.502518] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 1956.737688] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 1958.535241] Key type lgssc registered [ 1959.515173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1962.547204] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1962.561867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1963.774715] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1963.799709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1963.950349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.707796] Lustre: Failing over lustre-MDT0000 [ 1964.902919] Lustre: server umount lustre-MDT0000 complete [ 1970.636321] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 20:11:15 (1789603875) [ 1976.623552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1977.065436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1981.246338] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1988.917153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1988.989851] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1989.017973] Lustre: 64608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1989.212305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1993.376710] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 1999.553319] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2004.391092] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2010.779278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2011.015281] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2011.187851] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.558360] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2019.424263] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2023.867717] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2030.313340] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2036.353718] Lustre: server umount lustre-MDT0000 complete [ 2039.469323] LustreError: 64601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789603946 with bad export cookie 15207853785753345495 [ 2039.478588] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.489705] LustreError: 64601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2039.733280] Lustre: server umount lustre-MDT0001 complete [ 2042.555945] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2046.529954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2054.550228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.905320] Lustre: 63533:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789603946/real 1789603946] req@00000000134c885e x1876535660286336/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789603962 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2055.956904] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2060.255412] Lustre: 63534:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789603951/real 1789603951] req@000000005f82017d x1876535660286592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789603967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2060.288231] Lustre: 63534:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2064.353170] LustreError: 63531:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004decc5c0 x1876535660287616/t0(0) o250->MGC192.168.204.143@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2065.034342] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2065.139410] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2065.952311] Lustre: 63532:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789603956/real 1789603956] req@0000000086561aa9 x1876535660286848/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789603972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2065.999322] Lustre: 63532:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2066.026286] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2069.106854] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2071.012421] Lustre: 63535:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789603961/real 1789603961] req@00000000afaa3794 x1876535660287232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789603977 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2071.070631] Lustre: 63535:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2071.522820] LustreError: 68378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.561285] LustreError: 68378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2076.645972] LustreError: 68377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2076.663616] LustreError: 68377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2078.893894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2079.299288] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2083.655951] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2084.337024] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2091.204285] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2097.086907] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2103.801595] Lustre: server umount lustre-MDT0000 complete [ 2104.800588] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2104.827572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2104.842626] LustreError: 69391:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2104.845059] Lustre: Skipped 3 previous similar messages [ 2104.871493] LustreError: 69391:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2107.578988] LustreError: 64601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604014 with bad export cookie 15207853785753347770 [ 2107.580275] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2107.600259] LustreError: 64601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2107.936938] Lustre: server umount lustre-MDT0001 complete [ 2114.276861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2123.598198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2126.175377] Lustre: 63532:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789604016/real 1789604016] req@00000000a7002d73 x1876535660313088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789604032 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2126.215154] Lustre: 63532:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2126.224537] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.386620] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xd30d267187e86c1d [ 2132.396622] Lustre: MGC192.168.204.143@tcp: Connection restored to 0@lo (at 0@lo) [ 2132.959850] LustreError: 66458:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2132.976431] LustreError: 66458:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2133.073271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2133.123626] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2137.197550] LustreError: 71060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.228025] LustreError: 71060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2138.018631] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2140.127134] Lustre: 63533:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789604030/real 1789604030] req@0000000024c34b6c x1876535660313728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789604046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2140.172682] Lustre: 63533:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 2144.747641] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2146.160523] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.246724] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2151.913880] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2158.231296] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2163.141338] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2169.250838] Lustre: server umount lustre-MDT0000 complete [ 2172.384861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2172.389230] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2172.392697] LustreError: 66457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2172.401100] Lustre: Skipped 1 previous similar message [ 2172.421765] LustreError: 66457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 2172.976410] LustreError: 66463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604079 with bad export cookie 15207853785753349149 [ 2172.979371] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2172.985384] LustreError: 66463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2173.443711] Lustre: server umount lustre-MDT0001 complete [ 2179.447366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2182.553989] Lustre: DEBUG MARKER: files compared the same [ 2194.890575] Lustre: server umount lustre-OST0000 complete [ 2209.456614] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2211.842599] Key type lgssc unregistered [ 2212.127252] LNet: 74332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2212.137792] LNetError: 74332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2213.161733] LNet: Removed LNI 192.168.204.143@tcp [ 2213.937251] Key type .llcrypt unregistered [ 2213.942179] Key type ._llcrypt unregistered [ 2234.697731] Key type ._llcrypt registered [ 2234.705086] Key type .llcrypt registered [ 2234.816920] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2235.689208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2235.821177] alg: No test for adler32 (adler32-zlib) [ 2236.938481] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 2237.187073] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2238.871290] Key type lgssc registered [ 2239.916299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.106727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.137785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2248.475332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2252.246677] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug +malloc all [ 2258.914108] LustreError: 75739:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2258.930365] LustreError: 75739:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2260.870748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2261.186705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2265.733926] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug +malloc all [ 2273.365813] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2278.824180] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2285.777569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2286.030406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.867472] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug +malloc all [ 2294.266108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2298.332522] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2307.553358] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2307.564526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2307.593928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.600083] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2309.602272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.611378] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.629851] Lustre: Skipped 1 previous similar message [ 2312.030248] Lustre: server umount lustre-OST0000 complete [ 2317.791929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2317.801806] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2317.823853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.841975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.843903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2323.794616] Lustre: server umount lustre-MDT0000 complete [ 2324.961494] LustreError: 76479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.977863] LustreError: 76479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2327.124823] LustreError: 77584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604233 with bad export cookie 3700661495356256823 [ 2327.125046] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.138775] LustreError: 77584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2327.432724] Lustre: server umount lustre-MDT0001 complete [ 2337.202799] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2340.078345] Key type lgssc unregistered [ 2340.389128] LNet: 79271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2340.401363] LNetError: 79271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2340.419618] LNet: Removed LNI 192.168.204.143@tcp [ 2341.248389] Key type .llcrypt unregistered [ 2341.250813] Key type ._llcrypt unregistered [ 2363.622933] Key type ._llcrypt registered [ 2363.625919] Key type .llcrypt registered [ 2363.774791] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2364.456194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2364.512949] alg: No test for adler32 (adler32-zlib) [ 2365.688847] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 2366.035564] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2367.704673] Key type lgssc registered [ 2368.656989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.066238] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2376.096101] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2391.647448] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2396.767417] LustreError: 80657:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [ 2423.519321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2429.756819] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2437.185679] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2445.285250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2445.452316] Lustre: *** cfs_fail_loc=706, val=0*** [ 2445.902264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2445.997446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2450.327611] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2450.920433] LustreError: 80682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.944480] LustreError: 80682:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2456.039427] LustreError: 80683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.666158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.983734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2463.102243] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2470.041806] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2476.174703] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2479.589492] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2479.602800] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.612480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2480.616754] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2480.620905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2480.642056] Lustre: Skipped 1 previous similar message [ 2484.552389] Lustre: server umount lustre-OST0000 complete [ 2489.827096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2489.833332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2489.843097] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2494.519882] Lustre: server umount lustre-MDT0000 complete [ 2495.974885] LustreError: 81747:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2496.011969] LustreError: 81747:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2499.082235] LustreError: 80663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604405 with bad export cookie 11668476414035730611 [ 2499.092640] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2499.115802] LustreError: 80663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2499.632166] Lustre: server umount lustre-MDT0001 complete [ 2513.790108] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2516.705873] Key type lgssc unregistered [ 2517.078685] LNet: 84271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2517.087295] LNetError: 84271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2518.124383] LNet: Removed LNI 192.168.204.143@tcp [ 2518.938192] Key type .llcrypt unregistered [ 2518.941074] Key type ._llcrypt unregistered [ 2541.870343] Key type ._llcrypt registered [ 2541.872819] Key type .llcrypt registered [ 2541.965568] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2543.113206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2543.128878] alg: No test for adler32 (adler32-zlib) [ 2544.172493] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 2544.426814] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2546.095570] Key type lgssc registered [ 2547.341623] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2554.471566] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2554.488617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2559.509514] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2567.023428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2567.679688] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.352760] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2572.777582] LustreError: 86350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2572.804929] LustreError: 86350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2577.900056] LustreError: 86353:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.925289] LustreError: 86353:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2580.472856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2580.848221] LustreError: 86350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2580.866270] LustreError: 86350:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2580.886874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2586.093073] LustreError: 86354:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2586.120372] LustreError: 86354:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2587.028793] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2591.209122] LustreError: 86349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.245196] LustreError: 86349:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2595.194973] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2595.206098] Lustre: MGS: Client b2397d50-9d59-4343-9651-3d4210963ca8 (at 0@lo) reconnecting [ 2595.232290] LustreError: 86364:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2595.621291] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2595.760977] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2600.824357] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2616.301287] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2616.312525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2616.368763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2616.380786] Lustre: Skipped 1 previous similar message [ 2620.244580] Lustre: server umount lustre-OST0000 complete [ 2626.532171] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2626.532304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2626.559686] Lustre: Skipped 1 previous similar message [ 2626.567172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2626.582336] Lustre: Skipped 1 previous similar message [ 2630.642220] Lustre: server umount lustre-MDT0000 complete [ 2631.665157] LustreError: 87893:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.704241] LustreError: 87893:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2636.203465] Lustre: server umount lustre-MDT0001 complete [ 2640.991984] Lustre: server umount MGS complete [ 2656.486467] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 20:22:40 (1789604560) [ 2689.063296] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 2699.950926] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2745.586598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2750.767686] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2761.146763] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.423615] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2761.471391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2761.557457] Lustre: lustre-MDT0001: new disk, initializing [ 2761.649423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2761.693456] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2761.699777] Lustre: 93525:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2761.723050] LustreError: 93525:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2767.051835] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2778.782410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2779.038176] Lustre: lustre-OST0000: new disk, initializing [ 2779.041542] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2779.047442] Lustre: 94293:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2779.112936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.428352] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2788.960580] Lustre: 94751:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000001cf2726b x1876536276852480/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789604711 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2795.481868] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2803.701427] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2803.713666] Lustre: MGS: Client f435c8c9-2bb2-4034-85a0-91cd21f2f92b (at 0@lo) reconnecting [ 2803.746520] Lustre: 92795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2803.759941] LustreError: 93511:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2803.772286] LustreError: 93511:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2803.788204] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2803.857788] Lustre: lustre-MDT0000: new disk, initializing [ 2803.976024] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.011328] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2804.043756] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2805.089957] Lustre: 95365:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000823e47ba x1876536276859008/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1789604727 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2808.853309] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2810.272245] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2814.191129] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2814.198911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2814.348426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2840.036403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2840.049560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2840.069625] Lustre: Skipped 1 previous similar message [ 2840.081946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2842.088485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2842.099491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2842.121757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2845.791983] Lustre: server umount lustre-OST0000 complete [ 2853.346101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2853.346953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2853.370514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2853.390561] Lustre: Skipped 3 previous similar messages [ 2856.311526] Lustre: server umount lustre-MDT0000 complete [ 2858.465615] LustreError: 93497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2858.490455] LustreError: 93497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2867.511338] Lustre: server umount lustre-MDT0001 complete [ 2871.700479] Lustre: server umount MGS complete [ 2884.762261] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 20:26:29 (1789604789) [ 2905.847519] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 2909.092971] Key type lgssc unregistered [ 2909.390522] LNet: 97859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2909.413849] LNetError: 97859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2909.437053] LNet: Removed LNI 192.168.204.143@tcp [ 2910.621286] Key type .llcrypt unregistered [ 2910.623854] Key type ._llcrypt unregistered [ 2931.126083] Key type ._llcrypt registered [ 2931.128907] Key type .llcrypt registered [ 2931.234315] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 2932.423503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2932.509043] alg: No test for adler32 (adler32-zlib) [ 2933.762522] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 2934.040766] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 2935.815531] Key type lgssc registered [ 2936.968749] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2945.960763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2946.002037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2947.814408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2952.961714] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2957.283671] LustreError: 99077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2957.306935] LustreError: 99077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2962.402271] LustreError: 99076:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.400103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2964.060284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2969.161790] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 2978.303367] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2984.904867] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2993.064243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2993.660705] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3000.539914] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3008.071643] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3018.242506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3028.454278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3028.456940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3028.474410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3028.491168] Lustre: Skipped 1 previous similar message [ 3033.572404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3033.586153] Lustre: Skipped 2 previous similar messages [ 3038.690066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3038.704339] Lustre: Skipped 1 previous similar message [ 3040.223622] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3040.435250] Lustre: server umount lustre-OST0000 complete [ 3045.856449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3045.870624] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3045.894336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3048.930463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3050.836021] Lustre: server umount lustre-MDT0000 complete [ 3054.055574] LustreError: 99077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3054.082871] LustreError: 99077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3055.020656] LustreError: 99057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789604961 with bad export cookie 14359913758910038535 [ 3055.041201] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3055.403510] Lustre: server umount lustre-MDT0001 complete [ 3067.015690] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 3069.709950] Key type lgssc unregistered [ 3070.064938] LNet: 102631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3070.075296] LNetError: 102631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3070.097398] LNet: Removed LNI 192.168.204.143@tcp [ 3070.974397] Key type .llcrypt unregistered [ 3070.976572] Key type ._llcrypt unregistered [ 3088.155613] Key type ._llcrypt registered [ 3088.158377] Key type .llcrypt registered [ 3088.325375] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 3089.651471] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3089.677811] alg: No test for adler32 (adler32-zlib) [ 3090.784697] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 3091.052333] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 3092.791245] Key type lgssc registered [ 3094.296335] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3100.607874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3100.621180] LustreError: 103712:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3100.653749] LustreError: 103719:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3100.660538] LustreError: 103712:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4969 [ 3100.668061] LustreError: 103712:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3100.674763] LustreError: 103712:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3100.686264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.307260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3110.374074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3110.664118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3115.575654] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3121.167379] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing lsmod [ 3123.937166] LustreError: 104927:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3123.947353] LustreError: 104928:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3123.964947] LustreError: 104927:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4983 [ 3123.983750] LustreError: 104927:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3123.993776] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3124.004942] LustreError: 104927:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3124.509178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3129.837307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3129.848453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3129.858228] Lustre: Skipped 2 previous similar messages [ 3129.876145] Lustre: Skipped 2 previous similar messages [ 3133.531395] Lustre: server umount lustre-MDT0000 complete [ 3134.948121] LustreError: 103757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3134.973557] LustreError: 103757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 3135.981661] LustreError: 103757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.007945] LustreError: 103757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3137.915484] LustreError: 103736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605044 with bad export cookie 13770346711851509762 [ 3137.919987] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3137.924656] LustreError: 103736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3138.151599] LustreError: 104189:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b472c913 x1876536849354624/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3138.271763] Lustre: server umount lustre-MDT0001 complete [ 3152.817757] Lustre: server umount lustre-OST0000 complete [ 3162.730312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3163.239945] LustreError: 105992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.350335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3167.924718] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3168.749045] LustreError: 105993:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3173.859524] LustreError: 105992:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3177.088847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3177.542311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3182.671996] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3190.879116] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3198.445717] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3206.667766] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3207.129866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3209.213232] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3213.460366] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3221.122838] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3227.617283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.620481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3227.635523] Lustre: Skipped 1 previous similar message [ 3227.638087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.651154] Lustre: Skipped 1 previous similar message [ 3229.602813] Lustre: server umount lustre-OST0000 complete [ 3237.861917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.872623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3237.880767] Lustre: Skipped 1 previous similar message [ 3242.978037] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3242.984304] Lustre: Skipped 2 previous similar messages [ 3248.095201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3248.099146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.115457] Lustre: Skipped 1 previous similar message [ 3248.565811] Lustre: server umount lustre-MDT0000 complete [ 3253.019427] LustreError: 105973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605159 with bad export cookie 13770346711851510462 [ 3253.025649] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3253.043371] LustreError: 105973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3253.584252] Lustre: server umount lustre-MDT0001 complete [ 3266.835657] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 3269.936194] Key type lgssc unregistered [ 3270.315114] LNet: 109524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3270.322472] LNetError: 109524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3270.351227] LNet: Removed LNI 192.168.204.143@tcp [ 3271.512252] Key type .llcrypt unregistered [ 3271.514675] Key type ._llcrypt unregistered [ 3298.213939] Key type ._llcrypt registered [ 3298.216965] Key type .llcrypt registered [ 3298.404905] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 3299.761172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3299.983861] alg: No test for adler32 (adler32-zlib) [ 3301.290843] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 3301.604741] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 3303.359239] Key type lgssc registered [ 3304.948906] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3314.296328] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3314.324927] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3315.820818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3321.890397] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3325.477972] LustreError: 110939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3325.510869] LustreError: 110939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3330.541882] LustreError: 110938:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3331.586233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3331.962257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3338.132907] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3347.078360] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3354.170397] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3362.729916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3363.176888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.431943] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3378.837508] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3385.727506] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3395.113308] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3398.244690] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3406.306973] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3406.340290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3406.354495] Lustre: Skipped 1 previous similar message [ 3408.684780] Lustre: server umount lustre-OST0000 complete [ 3413.983942] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3413.988854] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3414.000452] Lustre: Skipped 1 previous similar message [ 3414.007965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.545052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3416.546688] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3416.578639] Lustre: Skipped 1 previous similar message [ 3418.795497] Lustre: server umount lustre-MDT0000 complete [ 3421.664459] LustreError: 110939:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.702959] LustreError: 110939:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3422.500757] LustreError: 110921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605329 with bad export cookie 6720563501000729926 [ 3422.501759] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3422.512587] LustreError: 110921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3422.961714] Lustre: server umount lustre-MDT0001 complete [ 3434.354717] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 3437.357144] Key type lgssc unregistered [ 3437.670790] LNet: 114600:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3437.690291] LNetError: 114600:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3437.720922] LNet: Removed LNI 192.168.204.143@tcp [ 3439.009442] Key type .llcrypt unregistered [ 3439.014143] Key type ._llcrypt unregistered [ 3461.550995] Key type ._llcrypt registered [ 3461.552719] Key type .llcrypt registered [ 3461.672161] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 3463.833739] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3463.939045] alg: No test for adler32 (adler32-zlib) [ 3465.233121] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 3465.538247] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 3467.319394] Key type lgssc registered [ 3468.793311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3479.535475] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3479.585430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3481.140861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3486.129862] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3490.284739] LustreError: 115820:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3490.310836] LustreError: 115820:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3495.401714] LustreError: 115821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3496.322030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3496.756064] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3501.389331] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3509.437974] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3516.048577] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3524.231837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3524.720602] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.787404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3532.927802] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3541.669927] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3551.714121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3551.736516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3551.759202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3552.737694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3552.739197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3556.715753] Lustre: server umount lustre-OST0000 complete [ 3562.981252] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3562.996713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3563.006038] Lustre: Skipped 1 previous similar message [ 3563.018995] Lustre: Skipped 2 previous similar messages [ 3568.097539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3568.109185] Lustre: Skipped 1 previous similar message [ 3573.218690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3573.224443] Lustre: Skipped 1 previous similar message [ 3575.264130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3575.524304] Lustre: server umount lustre-MDT0000 complete [ 3578.336776] LustreError: 115821:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3578.375579] LustreError: 115821:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3579.178848] LustreError: 116556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605485 with bad export cookie 9326044148825744455 [ 3579.183981] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3579.193216] LustreError: 116556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3579.545988] Lustre: server umount lustre-MDT0001 complete [ 3590.394727] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 3593.322894] Key type lgssc unregistered [ 3593.652551] LNet: 119355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3593.661335] LNetError: 119355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3593.681834] LNet: Removed LNI 192.168.204.143@tcp [ 3594.574816] Key type .llcrypt unregistered [ 3594.576700] Key type ._llcrypt unregistered [ 3621.981242] Key type ._llcrypt registered [ 3621.986191] Key type .llcrypt registered [ 3622.092450] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 3623.253139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3623.372235] alg: No test for adler32 (adler32-zlib) [ 3624.453196] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 3624.702529] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 3626.399242] Key type lgssc registered [ 3627.687605] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3635.441527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3635.452585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3636.903848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3641.257978] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3646.439369] LustreError: 120785:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3646.469589] LustreError: 120785:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3650.108802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3650.550075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3655.814606] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3664.798103] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3671.666832] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.377718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3680.659734] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3681.732847] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3688.366173] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3698.402922] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3707.827104] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3712.882251] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3712.893143] Lustre: Skipped 2 previous similar messages [ 3715.229717] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3715.286981] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3718.647668] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3718.661104] Lustre: Skipped 2 previous similar messages [ 3723.828641] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3723.841131] Lustre: Skipped 1 previous similar message [ 3728.372799] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.43@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3728.386942] Lustre: Skipped 2 previous similar messages [ 3731.535021] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.43@tcp 192.168.0.[2,10]@tcp [ 3738.158668] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.43/24@tcp in log lustre-MDT0000 [ 3738.178727] Lustre: Skipped 2 previous similar messages [ 3745.412902] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.43/24@tcp [ 3745.427211] Lustre: Skipped 1 previous similar message [ 3753.442909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3753.454054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3753.475263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.539388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3757.541542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3757.565456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3757.601508] Lustre: Skipped 1 previous similar message [ 3759.311814] Lustre: server umount lustre-OST0000 complete [ 3763.168103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3763.184564] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3763.214128] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.779482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.788113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3769.690527] Lustre: server umount lustre-MDT0000 complete [ 3772.900344] LustreError: 120781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3772.930770] LustreError: 120781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3773.927692] LustreError: 120764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605680 with bad export cookie 7280193141947423514 [ 3773.934600] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3773.950277] LustreError: 120764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3774.399136] Lustre: server umount lustre-MDT0001 complete [ 3788.612988] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 3791.754932] Key type lgssc unregistered [ 3792.037541] LNet: 125685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3792.060380] LNetError: 125685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3793.130618] LNet: Removed LNI 192.168.204.143@tcp [ 3793.921258] Key type .llcrypt unregistered [ 3793.927949] Key type ._llcrypt unregistered [ 3822.947860] Key type ._llcrypt registered [ 3822.952045] Key type .llcrypt registered [ 3823.074921] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 3823.863178] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3823.887777] alg: No test for adler32 (adler32-zlib) [ 3825.054652] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 3825.356765] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 3827.111619] Key type lgssc registered [ 3828.186256] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3836.407502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3836.421396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3837.930327] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3837.944146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3837.965181] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.43/24@tcp [ 3843.075561] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3848.171550] LustreError: 127242:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3848.204596] LustreError: 127242:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3853.178514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3853.282045] LustreError: 127241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3853.960510] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3854.018723] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.43/24@tcp [ 3854.104873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3859.697097] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3869.711877] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3876.508775] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3883.623541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3884.111848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3890.855478] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 3899.055725] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3907.783941] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3923.433516] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.449391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.449936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.453469] Lustre: Skipped 1 previous similar message [ 3926.269344] Lustre: server umount lustre-OST0000 complete [ 3931.104442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3931.111322] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3931.131782] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3931.138480] Lustre: Skipped 1 previous similar message [ 3933.668668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3933.670213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3933.702434] Lustre: Skipped 1 previous similar message [ 3937.417811] Lustre: server umount lustre-MDT0000 complete [ 3938.784160] LustreError: 127241:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.804704] LustreError: 127241:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3941.443978] LustreError: 129092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789605848 with bad export cookie 18181846186690475368 [ 3941.458359] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3941.462781] LustreError: 129092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3941.834141] Lustre: server umount lustre-MDT0001 complete [ 3955.051832] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 3958.518850] Key type lgssc unregistered [ 3958.812971] LNet: 130855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3958.821727] LNetError: 130855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3958.838230] LNet: Removed LNI 192.168.204.143@tcp [ 3959.731634] Key type .llcrypt unregistered [ 3959.733811] Key type ._llcrypt unregistered [ 3987.788721] Key type ._llcrypt registered [ 3987.794669] Key type .llcrypt registered [ 3987.957480] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 4003.428736] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4004.560406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4004.689725] alg: No test for adler32 (adler32-zlib) [ 4005.754902] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 4005.948505] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4007.575213] Key type lgssc registered [ 4008.611807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4044.578240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4044.619138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4045.907978] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4045.957080] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4046.117855] Lustre: lustre-MDT0000: new disk, initializing [ 4046.238264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4046.253644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4050.750725] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4061.682550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4061.798257] Lustre: 134267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4061.844269] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4061.848411] Lustre: Skipped 1 previous similar message [ 4061.936267] Lustre: lustre-MDT0001: new disk, initializing [ 4062.063249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4062.138036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4062.158531] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4066.895421] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4079.950463] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4086.189112] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4092.909608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4093.254419] Lustre: lustre-OST0000: new disk, initializing [ 4093.257971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4093.279751] Lustre: 136237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4093.390792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4100.359544] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4101.779210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4101.792157] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4101.974382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4111.157332] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4115.557992] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4115.780944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4120.115529] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4120.392641] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4122.598408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4122.607486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4122.608057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4122.614909] Lustre: Skipped 1 previous similar message [ 4127.718802] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4127.732937] Lustre: Skipped 2 previous similar messages [ 4128.547545] Lustre: server umount lustre-OST0000 complete [ 4132.842772] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4132.851406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4132.865876] Lustre: Skipped 1 previous similar message [ 4132.879772] Lustre: Skipped 1 previous similar message [ 4137.952960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4137.959498] Lustre: Skipped 1 previous similar message [ 4138.678976] Lustre: server umount lustre-MDT0000 complete [ 4142.543478] LustreError: 134258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606049 with bad export cookie 5394140236465925752 [ 4142.553618] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4142.558604] LustreError: 134258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4142.909091] Lustre: server umount lustre-MDT0001 complete [ 4151.189704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4151.601432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4155.742972] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4156.897538] LustreError: 138454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.928810] LustreError: 138454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4162.030285] LustreError: 138453:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4163.182296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4163.572602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4167.657567] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4174.948656] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4180.919899] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4188.615465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4188.898514] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4194.896421] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4203.756769] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4209.220788] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4209.494278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4214.956850] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4215.169779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4225.590154] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4225.884169] Lustre: lustre-OST0001: new disk, initializing [ 4225.889532] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4225.896129] Lustre: 141827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4226.035211] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4233.407541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4233.421600] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4233.612721] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4233.909333] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4243.479935] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4243.671244] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4248.377324] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4248.669996] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4254.545087] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b04e0dae000.ost_server_uuid 50 [ 4256.185745] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b04e0dae000.ost_server_uuid in IDLE state after 0 sec [ 4264.422083] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4264.423853] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4264.427722] LustreError: Skipped 1 previous similar message [ 4264.439577] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4264.448701] Lustre: Skipped 1 previous similar message [ 4269.871347] Lustre: server umount lustre-OST0001 complete [ 4274.146275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.160191] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4274.175982] Lustre: Skipped 1 previous similar message [ 4274.187821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.197561] Lustre: Skipped 2 previous similar messages [ 4274.662843] LustreError: 141342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4274.676317] LustreError: 141342:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4279.808472] Lustre: server umount lustre-OST0000 complete [ 4284.898858] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.922795] Lustre: Skipped 2 previous similar messages [ 4290.536719] Lustre: server umount lustre-MDT0000 complete [ 4295.069656] LustreError: 138433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606201 with bad export cookie 5394140236465926634 [ 4295.077816] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4295.088881] LustreError: 138433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4295.425912] Lustre: server umount lustre-MDT0001 complete [ 4316.968248] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4320.016782] Key type lgssc unregistered [ 4320.323755] LNet: 144831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4320.333568] LNetError: 144831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4320.352799] LNet: Removed LNI 192.168.204.143@tcp [ 4321.247995] Key type .llcrypt unregistered [ 4321.250086] Key type ._llcrypt unregistered [ 4363.258953] Key type ._llcrypt registered [ 4363.262592] Key type .llcrypt registered [ 4363.406992] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 4379.297070] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4380.751021] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4380.804405] alg: No test for adler32 (adler32-zlib) [ 4381.877627] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 4382.226918] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4383.943257] Key type lgssc registered [ 4385.405349] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4420.529781] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4420.561860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4421.783710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4421.810128] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4421.873416] Lustre: lustre-MDT0000: new disk, initializing [ 4421.934156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4421.952650] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4426.634582] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4436.982296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4437.099679] Lustre: 148983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4437.152872] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4437.158873] Lustre: Skipped 1 previous similar message [ 4437.257911] Lustre: lustre-MDT0001: new disk, initializing [ 4437.353535] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4437.398716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4437.413252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4441.503644] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4451.449958] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4457.523909] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4464.285948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4464.608573] Lustre: lustre-OST0000: new disk, initializing [ 4464.612895] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4464.620549] Lustre: 150950:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4464.701171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4471.941268] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4472.417807] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4472.432042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4472.546273] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4480.890875] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4485.589187] Lustre: Failing over lustre-OST0000 [ 4485.702926] Lustre: server umount lustre-OST0000 complete [ 4503.512213] Lustre: 147206:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606394/real 1789606394] req@000000009aabb1e0 x1876538203080320/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1789606410 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4503.543719] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4503.873459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.174708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.191287] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4505.190172] Lustre: 147207:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606394/real 1789606394] req@000000000fcb85a1 x1876538203080192/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789606410 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4505.223189] Lustre: 147207:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4509.279105] Lustre: 147204:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606400/real 1789606400] req@0000000090026280 x1876538203081472/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789606416 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4510.348310] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4513.763829] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4514.142846] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4514.149419] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4514.218674] Lustre: Failing over lustre-MDT0000 [ 4514.272355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4514.284131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4514.301597] Lustre: Skipped 1 previous similar message [ 4514.312883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4514.563474] Lustre: server umount lustre-MDT0000 complete [ 4515.364152] Lustre: 147204:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789606405/real 1789606405] req@00000000c80092e4 x1876538203082496/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1789606421 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4515.380513] Lustre: 147204:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4518.893052] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4518.924763] Lustre: Skipped 1 previous similar message [ 4524.011526] LustreError: 148990:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4524.043059] LustreError: 148990:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4529.131105] LustreError: 148995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4529.155142] LustreError: 148995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4534.234061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4534.254233] LustreError: 148994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4534.282123] LustreError: 148994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4534.394084] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4534.580790] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.43@tcp (not set up) [ 4534.801879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4534.878176] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4539.706767] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4539.885729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4539.905399] Lustre: Skipped 1 previous similar message [ 4539.924387] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4539.968736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4539.971703] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4550.116622] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4550.117094] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4550.139912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4550.145513] Lustre: Skipped 1 previous similar message [ 4555.243101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4555.265954] Lustre: Skipped 2 previous similar messages [ 4555.610913] Lustre: server umount lustre-OST0000 complete [ 4560.353735] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4560.361423] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4560.393184] Lustre: Skipped 1 previous similar message [ 4560.406451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4560.417145] Lustre: Skipped 1 previous similar message [ 4566.823269] Lustre: server umount lustre-MDT0000 complete [ 4570.602196] LustreError: 148994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4570.631822] LustreError: 148994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4570.738351] LustreError: 148973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606477 with bad export cookie 17749087161520561991 [ 4570.743264] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4570.753831] LustreError: 148973:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4571.180867] Lustre: server umount lustre-MDT0001 complete [ 4583.808859] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 4586.453768] Key type lgssc unregistered [ 4586.766796] LNet: 154601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4586.785251] LNetError: 154601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4586.795883] LNet: Removed LNI 192.168.204.143@tcp [ 4587.709814] Key type .llcrypt unregistered [ 4587.711581] Key type ._llcrypt unregistered [ 4623.580895] Key type ._llcrypt registered [ 4623.583394] Key type .llcrypt registered [ 4623.708531] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4624.812038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4624.862189] alg: No test for adler32 (adler32-zlib) [ 4625.897366] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 4626.109876] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 4627.767241] Key type lgssc registered [ 4628.869991] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4637.365559] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4637.415240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4639.122212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4644.772560] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4648.423219] LustreError: 156121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4648.438229] LustreError: 156121:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4653.540421] LustreError: 156121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4655.864602] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4658.661127] LustreError: 156122:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.222034] LustreError: 156150:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4660.245202] LustreError: 156150:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4660.257479] LustreError: 156150:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4660.930864] Lustre: server umount lustre-MDT0000 complete [ 4678.991995] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 4689.819696] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 4724.299237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4724.527545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4724.559875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4724.625683] Lustre: lustre-MDT0000: new disk, initializing [ 4724.719316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4724.735667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4729.163264] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4739.782553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4739.915258] Lustre: 159457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4739.981885] LustreError: 159467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4740.007985] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4740.019076] Lustre: Skipped 1 previous similar message [ 4740.084231] Lustre: lustre-MDT0001: new disk, initializing [ 4740.189110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4740.224538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4740.234497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4744.571978] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4754.463564] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4760.414892] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4767.069332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4767.320057] Lustre: lustre-OST0000: new disk, initializing [ 4767.328420] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4767.334846] Lustre: 161424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4767.441357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.893758] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 4774.956654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4774.973384] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4775.079239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4781.908597] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4786.572086] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4788.259750] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:58:13 (1789606693) [ 5036.512995] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5036.515976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5036.527601] Lustre: Skipped 2 previous similar messages [ 5036.540552] Lustre: Skipped 2 previous similar messages [ 5040.863449] Lustre: server umount lustre-MDT0000 complete [ 5041.632524] LustreError: 159463:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5041.650112] LustreError: 159463:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5045.580544] LustreError: 160251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789606952 with bad export cookie 242165950400919985 [ 5045.588528] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5045.603392] LustreError: 160251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5046.755490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.776876] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5050.797314] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5050.803669] LustreError: 162095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5050.825685] LustreError: 162095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5052.150388] Lustre: server umount lustre-MDT0001 complete [ 5062.513283] Lustre: server umount lustre-OST0000 complete [ 5072.316502] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 5082.084801] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5120.419350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5120.739356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5120.802261] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5120.968884] Lustre: lustre-MDT0000: new disk, initializing [ 5121.121544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5121.151531] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5126.399279] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5138.174599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5138.276925] Lustre: 166098:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5138.350684] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5138.357704] Lustre: Skipped 1 previous similar message [ 5138.431635] Lustre: lustre-MDT0001: new disk, initializing [ 5138.501841] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5138.538244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5138.555594] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5143.292345] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5154.439973] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5159.429687] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5166.004868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5166.406785] Lustre: lustre-OST0000: new disk, initializing [ 5166.410198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5166.415755] Lustre: 168064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5166.510437] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5173.279833] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5176.375135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5176.388230] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5176.516439] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5182.549412] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5187.217683] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5187.462274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5192.408989] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5192.595207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5197.280946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5197.294637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5197.299977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5197.307319] Lustre: Skipped 1 previous similar message [ 5197.326029] LustreError: Skipped 1 previous similar message [ 5200.841971] Lustre: server umount lustre-OST0000 complete [ 5204.964409] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5204.973155] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.989948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.997093] Lustre: Skipped 1 previous similar message [ 5210.423720] Lustre: server umount lustre-MDT0000 complete [ 5212.642127] LustreError: 166103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5212.663831] LustreError: 166103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5214.469661] LustreError: 166090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607121 with bad export cookie 242165950401078815 [ 5214.471690] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5214.483693] LustreError: 166090:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5214.747985] Lustre: server umount lustre-MDT0001 complete [ 5223.748809] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:05:28 (1789607128) [ 5239.673873] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 5248.019952] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5281.388416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.614845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5281.644839] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5281.720843] Lustre: lustre-MDT0000: new disk, initializing [ 5281.794838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5281.813362] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5286.818287] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5297.590521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.688587] Lustre: 172823:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5297.923965] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5302.252573] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5313.410523] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5321.017498] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5330.514867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5331.027874] Lustre: lustre-OST0000: new disk, initializing [ 5331.032770] Lustre: Skipped 1 previous similar message [ 5331.042791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5331.055507] Lustre: Skipped 2 previous similar messages [ 5331.064991] Lustre: 174797:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5337.727592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5337.736130] Lustre: Skipped 1 previous similar message [ 5337.747706] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5337.844456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5339.879354] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5352.933626] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5363.684276] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5363.685656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.690000] LustreError: Skipped 1 previous similar message [ 5363.693050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5363.701915] Lustre: Skipped 2 previous similar messages [ 5363.725490] Lustre: Skipped 3 previous similar messages [ 5366.916770] Lustre: server umount lustre-OST0000 complete [ 5373.922666] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.948358] Lustre: Skipped 1 previous similar message [ 5377.602310] Lustre: server umount lustre-MDT0000 complete [ 5379.045836] LustreError: 174485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5379.076623] LustreError: 174485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5382.190780] LustreError: 172814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607288 with bad export cookie 242165950401079697 [ 5382.194038] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5382.203065] LustreError: 172814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5382.781911] Lustre: server umount lustre-MDT0001 complete [ 5396.457573] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 21:08:20 (1789607300) [ 5415.824224] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 5426.342711] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5468.603833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5468.974996] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5469.033661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5469.109293] Lustre: lustre-MDT0000: new disk, initializing [ 5469.264117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5469.283939] Lustre: Skipped 2 previous similar messages [ 5469.307616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5475.092584] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5485.707878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5485.877478] Lustre: 179126:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5486.220821] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5490.265951] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5498.744797] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5504.089233] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5512.152419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5512.545303] Lustre: 181108:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5519.380633] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5519.937495] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5520.118667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5531.137493] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5540.837630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5540.838304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5540.848806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5540.864625] Lustre: Skipped 1 previous similar message [ 5540.883583] Lustre: Skipped 3 previous similar messages [ 5544.874170] Lustre: server umount lustre-OST0000 complete [ 5555.672668] Lustre: server umount lustre-MDT0000 complete [ 5556.204993] LustreError: 179138:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.238437] LustreError: 179138:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5559.800433] LustreError: 179119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607466 with bad export cookie 242165950401080782 [ 5559.805569] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5559.820242] LustreError: 179119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5560.180369] Lustre: server umount lustre-MDT0001 complete [ 5572.765235] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5576.059839] Key type lgssc unregistered [ 5576.294626] LNet: 182965:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5576.302645] LNetError: 182965:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5577.317689] LNet: Removed LNI 192.168.204.143@tcp [ 5578.121093] Key type .llcrypt unregistered [ 5578.128390] Key type ._llcrypt unregistered [ 5604.630668] Key type ._llcrypt registered [ 5604.634543] Key type .llcrypt registered [ 5604.769913] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5606.309124] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5606.368599] alg: No test for adler32 (adler32-zlib) [ 5607.662895] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 5607.953639] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 5609.648319] Key type lgssc registered [ 5610.821946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5620.428919] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5620.443722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5622.180588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5628.270084] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5631.463212] LustreError: 184378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5631.501245] LustreError: 184378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5636.582560] LustreError: 184379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5640.232857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5640.717369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5647.407949] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5657.663934] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5663.699558] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5671.508705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5671.959932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5679.144783] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5686.841456] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5691.468617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5715.943382] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5715.950842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5715.969562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.964891] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5716.973282] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5717.014134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5718.425749] Lustre: server umount lustre-OST0000 complete [ 5727.206732] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5727.208083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5727.221846] Lustre: Skipped 1 previous similar message [ 5727.240309] Lustre: Skipped 2 previous similar messages [ 5729.280497] Lustre: server umount lustre-MDT0000 complete [ 5732.324736] LustreError: 185622:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5732.368237] LustreError: 185622:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5733.438345] LustreError: 184360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607640 with bad export cookie 6871500182354598568 [ 5733.440764] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5733.448211] LustreError: 184360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5733.927808] Lustre: server umount lustre-MDT0001 complete [ 5746.247296] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 5749.731204] Key type lgssc unregistered [ 5750.059566] LNet: 187936:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5750.068760] LNetError: 187936:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5750.100128] LNet: Removed LNI 192.168.204.143@tcp [ 5751.231741] Key type .llcrypt unregistered [ 5751.234237] Key type ._llcrypt unregistered [ 5781.872884] Key type ._llcrypt registered [ 5781.876185] Key type .llcrypt registered [ 5781.973931] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 5782.962656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5783.144312] alg: No test for adler32 (adler32-zlib) [ 5784.233820] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 5784.626760] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 5786.303814] Key type lgssc registered [ 5787.578557] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5795.607783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5795.650399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5797.155194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5802.071029] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5806.580776] LustreError: 189351:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5806.600800] LustreError: 189351:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5810.615589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5810.977615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5815.343923] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5822.900638] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5830.250470] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5837.236461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5837.683397] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5841.803051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5844.208620] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5852.609555] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5864.800177] Lustre: lustre-OST0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 5866.470787] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5866.473473] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.509906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5866.527987] Lustre: Skipped 1 previous similar message [ 5869.915911] Lustre: lustre-OST0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 5869.938505] Lustre: Skipped 1 previous similar message [ 5875.005712] Lustre: lustre-OST0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 5875.017771] Lustre: Skipped 2 previous similar messages [ 5877.216276] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5877.674726] Lustre: server umount lustre-OST0000 complete [ 5885.185606] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b04c781a800.ost_server_uuid 50 [ 5887.860560] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5903.329092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5903.337375] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5903.346924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5903.358150] Lustre: Skipped 2 previous similar messages [ 5906.404735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5908.882428] Lustre: server umount lustre-MDT0000 complete [ 5911.529094] LustreError: 189345:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5911.557823] LustreError: 189345:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5912.872213] LustreError: 189331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607819 with bad export cookie 2497313966128066795 [ 5912.877604] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5912.887764] LustreError: 189331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5913.496215] Lustre: server umount lustre-MDT0001 complete [ 5924.266700] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 21:17:08 (1789607828) [ 5931.756429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5932.307791] LustreError: 193276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5932.444376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5936.854986] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5937.641793] LustreError: 193277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5942.763100] LustreError: 193276:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5946.067730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.531138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5952.204468] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5960.855329] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5967.557845] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5975.930080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.515509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5977.543184] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5983.869671] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 5992.752933] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6001.575169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6001.791810] Lustre: lustre-OST0001: new disk, initializing [ 6001.795509] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6001.800286] Lustre: 196120:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6001.896201] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6008.966529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6008.985566] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6009.184430] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6009.577040] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6020.541578] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6023.650029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6023.666510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6023.691281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6023.700593] Lustre: Skipped 2 previous similar messages [ 6029.831820] LustreError: 195100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6029.989811] Lustre: server umount lustre-OST0000 complete [ 6039.052696] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6039.334243] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6045.098185] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6045.413766] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6047.028321] LustreError: 195098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6047.060398] LustreError: 195098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 6060.522284] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6060.530743] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6060.533673] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6060.542836] Lustre: Skipped 2 previous similar messages [ 6060.561881] Lustre: Skipped 3 previous similar messages [ 6066.230488] Lustre: server umount lustre-OST0001 complete [ 6070.753140] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6076.901766] Lustre: server umount lustre-MDT0000 complete [ 6080.966904] LustreError: 195121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789607987 with bad export cookie 2497313966128067859 [ 6080.978056] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6080.979831] LustreError: 195121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6081.404730] Lustre: server umount lustre-MDT0001 complete [ 6108.594731] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 21:20:13 (1789608013) [ 6117.403936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6117.861318] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6117.914307] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6118.103520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6123.485141] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6133.748077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6133.931820] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6133.978523] Lustre: 199557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6139.457333] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6148.511090] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6155.567258] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6163.639725] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6163.809851] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6172.690469] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6174.411379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6182.084611] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6190.201831] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6190.310821] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6190.686096] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6190.696854] Lustre: Skipped 2 previous similar messages [ 6198.994646] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6207.784541] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6213.983476] Lustre: lustre-OST0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 6214.006248] Lustre: Skipped 6 previous similar messages [ 6214.140574] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6214.159046] Lustre: Skipped 2 previous similar messages [ 6225.375388] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6225.611307] Lustre: server umount lustre-OST0000 complete [ 6229.299277] LustreError: 201411:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6229.322071] LustreError: 201411:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 6244.836804] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6244.846050] LustreError: Skipped 2 previous similar messages [ 6248.884132] Lustre: server umount lustre-OST0001 complete [ 6255.073158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6255.101715] Lustre: Skipped 3 previous similar messages [ 6267.360798] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6267.562112] Lustre: server umount lustre-MDT0000 complete [ 6271.760430] LustreError: 199550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608178 with bad export cookie 2497313966128069273 [ 6271.765794] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6271.777489] LustreError: 199550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6272.242341] Lustre: server umount lustre-MDT0001 complete [ 6305.011937] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 21:23:29 (1789608209) [ 6324.636144] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 6335.344710] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 6376.647618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.041973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6377.102981] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6377.200325] Lustre: lustre-MDT0000: new disk, initializing [ 6377.293684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6377.327581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6383.073523] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6395.564141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.757655] Lustre: 207660:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6395.821671] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6395.826122] Lustre: Skipped 1 previous similar message [ 6395.930377] Lustre: lustre-MDT0001: new disk, initializing [ 6396.151061] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6396.169452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6402.095977] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6413.381521] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6419.832807] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6427.257371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6427.640306] Lustre: lustre-OST0000: new disk, initializing [ 6427.645409] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6427.654225] Lustre: 209626:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6434.778771] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6437.962082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6437.976894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6438.164470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6445.154359] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6450.893740] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6451.139438] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6456.307784] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6456.522642] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6458.851520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6458.852492] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6458.866683] LustreError: Skipped 1 previous similar message [ 6458.867376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6458.867380] Lustre: Skipped 16 previous similar messages [ 6458.916175] Lustre: Skipped 1 previous similar message [ 6464.978423] Lustre: server umount lustre-OST0000 complete [ 6475.681171] Lustre: server umount lustre-MDT0000 complete [ 6479.351130] LustreError: 207673:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.403432] LustreError: 207673:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6480.493537] LustreError: 207653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608387 with bad export cookie 2497313966128070841 [ 6480.508863] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6480.516748] LustreError: 207653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6481.205647] Lustre: server umount lustre-MDT0001 complete [ 6491.817354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6498.505864] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6509.536349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.010931] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6524.282773] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6531.173143] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6539.918657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6547.759804] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6555.921809] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6561.339746] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6565.049458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 6570.378677] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6570.679162] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6574.063340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6574.078039] LustreError: Skipped 1 previous similar message [ 6578.899934] Lustre: server umount lustre-OST0000 complete [ 6585.763435] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6590.993314] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6621.075364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6628.675837] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6637.376776] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6638.891148] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.924199] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6638.930753] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6638.965867] Lustre: Skipped 1 previous similar message [ 6639.404234] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6644.199436] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6644.203572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6644.207847] LustreError: Skipped 1 previous similar message [ 6644.221460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6644.224390] Lustre: Skipped 5 previous similar messages [ 6644.248333] Lustre: Skipped 9 previous similar messages [ 6649.923319] Lustre: server umount lustre-OST0000 complete [ 6660.747563] Lustre: server umount lustre-MDT0000 complete [ 6664.672876] LustreError: 213684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6664.704482] LustreError: 213684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6664.898918] LustreError: 216677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608571 with bad export cookie 2497313966128071723 [ 6664.908357] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6664.915929] LustreError: 216677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6665.589707] Lustre: server umount lustre-MDT0001 complete [ 6676.666290] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 21:29:41 (1789608581) [ 6685.581370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6686.284971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6686.291286] Lustre: Skipped 6 previous similar messages [ 6692.370381] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6701.632566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6706.731443] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6715.655732] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6722.558119] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6730.453832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6737.912781] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6746.685916] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6751.502945] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6751.784234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6759.554677] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6759.947718] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6768.291524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6768.582762] Lustre: lustre-OST0001: new disk, initializing [ 6768.590828] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6768.596815] Lustre: 221206:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6776.249678] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6778.518377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6778.531232] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6778.592878] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6788.100302] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6793.326400] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6793.562908] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6798.988104] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6799.243178] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6804.456154] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6807.372709] Lustre: server umount lustre-OST0001 complete [ 6814.715790] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6814.975683] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6819.922140] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6820.136324] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6851.190253] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6856.708571] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6856.726334] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6856.728942] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6859.940518] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6869.864359] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6871.943896] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6880.644910] Lustre: server umount lustre-OST0001 complete [ 6901.216935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.221189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6901.233129] Lustre: Skipped 7 previous similar messages [ 6901.260274] Lustre: Skipped 12 previous similar messages [ 6904.303692] Lustre: server umount lustre-OST0000 complete [ 6914.628183] Lustre: server umount lustre-MDT0000 complete [ 6919.508400] LustreError: 217818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789608826 with bad export cookie 2497313966128072689 [ 6919.528293] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6926.112974] Lustre: server umount lustre-MDT0001 complete [ 6953.377739] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 21:34:18 (1789608858) [ 6960.888855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6961.345721] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6961.426137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6967.176283] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6977.189790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6977.377773] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6977.440906] Lustre: 226458:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6982.986948] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 6991.491371] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6998.561428] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7006.541418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7006.914223] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7014.598356] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7022.371407] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7032.524813] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7032.691135] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7040.790702] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7048.921509] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7050.477914] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7055.070749] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7055.292424] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7060.374992] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7060.622372] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7066.443905] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b04e064c000.ost_server_uuid 50 [ 7068.036341] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b04e064c000.ost_server_uuid in IDLE state after 0 sec [ 7069.154107] Lustre: Permanently deactivating lustre-OST0001 [ 7069.178234] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7072.674480] Lustre: Permanently reactivating lustre-OST0001 [ 7079.428246] Lustre: server umount lustre-OST0001 complete [ 7080.428475] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7080.439773] LustreError: 228303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7080.446028] LustreError: Skipped 6 previous similar messages [ 7080.472413] LustreError: 228303:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 7098.336315] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7098.626171] Lustre: server umount lustre-OST0000 complete [ 7109.154102] Lustre: server umount lustre-MDT0000 complete [ 7113.194342] LustreError: 227198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609019 with bad export cookie 2497313966128073977 [ 7113.201096] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.214981] LustreError: 227198:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7113.811211] Lustre: server umount lustre-MDT0001 complete [ 7146.634192] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:37:31 (1789609051) [ 7155.742898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7156.187251] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7156.223322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7156.232413] Lustre: Skipped 5 previous similar messages [ 7161.589776] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7171.658425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7171.825320] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7171.930658] Lustre: 233155:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7176.698457] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7183.848809] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7190.250982] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7197.514831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7197.782597] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7197.832695] Lustre: Permanently deactivating lustre-OST0000 [ 7197.845993] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7205.362915] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7206.978197] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7214.706700] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7222.336925] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.394557] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7222.611341] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7222.616672] Lustre: Skipped 11 previous similar messages [ 7229.810814] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7232.658678] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7236.948127] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7238.741288] Lustre: Permanently reactivating lustre-OST0000 [ 7238.767564] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7238.773270] Lustre: Skipped 2 previous similar messages [ 7242.112527] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7242.137254] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 35 seconds. I think it's dead, and I am evicting it. exp 00000000beceb4d5, cur 1789609148 deadline 1789609143 last 1789609113 [ 7242.212048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7262.458447] Lustre: server umount lustre-OST0001 complete [ 7282.145269] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7282.330546] Lustre: server umount lustre-OST0000 complete [ 7301.087190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7301.469846] Lustre: server umount lustre-MDT0000 complete [ 7306.412893] LustreError: 233148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609213 with bad export cookie 2497313966128075650 [ 7306.413290] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7306.430758] LustreError: 233148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7307.225232] Lustre: server umount lustre-MDT0001 complete [ 7319.772034] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 7323.169655] Key type lgssc unregistered [ 7323.537543] LNet: 238040:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7323.549597] LNetError: 238040:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7323.570761] LNet: Removed LNI 192.168.204.143@tcp [ 7324.620615] Key type .llcrypt unregistered [ 7324.625946] Key type ._llcrypt unregistered [ 7350.107228] Key type ._llcrypt registered [ 7350.116021] Key type .llcrypt registered [ 7350.272631] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 7351.828760] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7351.877713] alg: No test for adler32 (adler32-zlib) [ 7353.181995] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 7353.483924] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 7355.167232] Key type lgssc registered [ 7356.930174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7368.817208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7368.849132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7370.671522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.211853] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7380.975938] LustreError: 239506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7380.992531] LustreError: 239506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7385.485921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.614402] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7385.639838] Lustre: 239493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7385.649722] Lustre: Permanently deactivating lustre-MDT0001 [ 7385.662330] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7386.001758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7387.170310] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7387.177648] LustreError: 239536:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7387.194259] LustreError: 239536:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7387.210589] LustreError: 239536:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7391.092789] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7399.751713] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7406.744771] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7414.273489] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7414.763114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7415.805356] LustreError: 241347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7415.826249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7420.906737] LustreError: 241347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7421.955988] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7426.039626] LustreError: 241350:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.154964] LustreError: 241347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7431.189567] LustreError: 241347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7431.221393] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7432.054931] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.888942] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7440.128783] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7445.524853] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7445.533293] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7448.158310] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7456.260970] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7459.213734] Lustre: Permanently reactivating lustre-MDT0001 [ 7459.243948] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7459.254317] Lustre: Skipped 2 previous similar messages [ 7465.327177] Lustre: lustre-MDT0001: Client a9e8e936-6c1a-40c0-b4cc-59d92cbc22dd (at 192.168.204.43@tcp) reconnecting [ 7471.775516] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7473.323140] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7479.054884] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7479.293969] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7484.376511] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7484.613253] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7487.592625] Lustre: Permanently deactivating lustre-MDT0001 [ 7487.616670] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7487.623827] Lustre: Skipped 2 previous similar messages [ 7497.918386] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7506.916754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7506.919869] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7506.943562] Lustre: Skipped 2 previous similar messages [ 7506.954015] Lustre: Skipped 1 previous similar message [ 7510.423308] Lustre: server umount lustre-MDT0000 complete [ 7512.035884] LustreError: 239505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7512.063870] LustreError: 239505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 7515.051332] LustreError: 239487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609421 with bad export cookie 4917919994829817675 [ 7515.056656] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7515.071819] LustreError: 239487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7515.544482] Lustre: server umount lustre-MDT0001 complete [ 7522.319564] Lustre: server umount lustre-OST0000 complete [ 7534.522419] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:43:58 (1789609438) [ 7547.416453] Lustre: server umount lustre-OST0001 complete [ 7555.589491] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 7568.704856] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 7611.350584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7611.691971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7611.702476] Lustre: Skipped 2 previous similar messages [ 7611.745293] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7611.849325] Lustre: lustre-MDT0000: new disk, initializing [ 7611.948643] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7611.964461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7617.008147] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7631.030536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7631.284489] Lustre: 247778:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7631.384519] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7631.394304] Lustre: Skipped 1 previous similar message [ 7631.521675] Lustre: lustre-MDT0001: new disk, initializing [ 7631.771215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7631.817771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7631.839399] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7638.518266] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7652.604896] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7659.369604] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7666.661824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7667.048432] Lustre: lustre-OST0000: new disk, initializing [ 7667.057995] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7667.076988] Lustre: 249747:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7667.176729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7672.489694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7672.517223] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7672.683380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7674.972706] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7686.733702] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7688.793860] LustreError: 247786:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7698.616081] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7698.796378] Lustre: lustre-OST0001: new disk, initializing [ 7698.800147] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7698.811501] Lustre: 250864:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7698.887170] LustreError: 247786:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7698.945853] LustreError: 249728:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7705.060893] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7705.720390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7705.727310] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7705.880199] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7709.011565] LustreError: 249728:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7715.507728] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7717.826539] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7721.441366] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7721.441400] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7721.441426] Lustre: Skipped 1 previous similar message [ 7721.441930] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7721.469419] LustreError: Skipped 1 previous similar message [ 7721.511495] Lustre: Skipped 2 previous similar messages [ 7725.755577] Lustre: server umount lustre-OST0001 complete [ 7726.566573] LustreError: 249996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7726.587995] LustreError: 249996:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7744.482198] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7744.482919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7744.504042] LustreError: Skipped 1 previous similar message [ 7744.506275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7744.532359] Lustre: Skipped 2 previous similar messages [ 7747.042971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7747.057389] Lustre: Skipped 2 previous similar messages [ 7752.164579] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7752.171328] Lustre: Skipped 1 previous similar message [ 7757.279404] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7757.424789] Lustre: server umount lustre-OST0000 complete [ 7762.401899] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7762.421033] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7762.428249] Lustre: Skipped 3 previous similar messages [ 7768.049587] Lustre: server umount lustre-MDT0000 complete [ 7772.641984] LustreError: 247771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609679 with bad export cookie 4917919994829820489 [ 7772.646152] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7772.655682] LustreError: 247771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7772.655792] LustreError: 238715:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c04f7819 x1876541319573376/t0(0) o38->lustre-MDT0000-osp-MDT0001@0@lo:24/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7779.265659] Lustre: server umount lustre-MDT0001 complete [ 7793.902881] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 7797.938385] Key type lgssc unregistered [ 7798.336891] LNet: 252897:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7798.343972] LNetError: 252897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7798.356875] LNet: Removed LNI 192.168.204.143@tcp [ 7799.454107] Key type .llcrypt unregistered [ 7799.455981] Key type ._llcrypt unregistered [ 7842.319514] Key type ._llcrypt registered [ 7842.322363] Key type .llcrypt registered [ 7842.421330] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 7843.997638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7844.059515] alg: No test for adler32 (adler32-zlib) [ 7845.146840] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 7845.458326] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 7847.207360] Key type lgssc registered [ 7848.564817] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7856.740971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7856.775488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7858.086944] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7858.110528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7858.283759] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.420797] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7872.948693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7873.035920] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7873.091855] Lustre: 255102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7873.424197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7878.349171] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7887.152785] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7893.691500] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7901.157943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7901.334154] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7901.580332] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7907.430203] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7910.549684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7915.540386] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7967.201051] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7967.205558] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7967.211417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7967.246935] Lustre: Skipped 1 previous similar message [ 7972.326185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7972.333423] Lustre: Skipped 2 previous similar messages [ 7973.354871] Lustre: server umount lustre-OST0000 complete [ 7977.827371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7990.508599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7990.935907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7997.257109] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 7998.128633] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7998.138481] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7998.140281] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7998.168242] Lustre: Skipped 1 previous similar message [ 8004.618568] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8044.513461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8044.520361] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8044.551198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8044.588502] Lustre: Skipped 1 previous similar message [ 8049.644705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8049.656069] Lustre: Skipped 1 previous similar message [ 8050.305552] Lustre: server umount lustre-OST0000 complete [ 8054.754975] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8054.769440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8054.786109] Lustre: Skipped 2 previous similar messages [ 8054.811701] Lustre: Skipped 1 previous similar message [ 8060.239809] Lustre: server umount lustre-MDT0000 complete [ 8064.656648] LustreError: 255093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789609971 with bad export cookie 8397048177537290811 [ 8064.668172] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8064.685832] LustreError: 255093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8065.167206] Lustre: server umount lustre-MDT0001 complete [ 8075.795122] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 8078.333677] Key type lgssc unregistered [ 8078.605781] LNet: 260526:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8078.611883] LNetError: 260526:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8078.631733] LNet: Removed LNI 192.168.204.143@tcp [ 8079.276910] Key type .llcrypt unregistered [ 8079.279084] Key type ._llcrypt unregistered [ 8104.416212] Key type ._llcrypt registered [ 8104.417975] Key type .llcrypt registered [ 8104.539476] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 8105.836474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8105.977743] alg: No test for adler32 (adler32-zlib) [ 8107.325007] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8107.729333] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 8109.391296] Key type lgssc registered [ 8110.654772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8118.911653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8118.938442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8120.467659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8125.408745] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8131.043996] LustreError: 261940:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8131.069151] LustreError: 261940:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8135.136629] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8135.447239] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8140.257330] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8148.121626] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8154.212459] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8161.352875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8161.791521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8165.913979] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8169.052332] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8176.184262] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8205.799692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8205.821827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8205.831438] Lustre: Skipped 1 previous similar message [ 8205.850771] Lustre: Skipped 1 previous similar message [ 8210.915633] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8210.929142] Lustre: Skipped 1 previous similar message [ 8211.274756] Lustre: server umount lustre-OST0000 complete [ 8216.035412] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8216.037049] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8216.061177] Lustre: Skipped 1 previous similar message [ 8216.072735] Lustre: Skipped 1 previous similar message [ 8221.154902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8221.168458] Lustre: Skipped 1 previous similar message [ 8226.272232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8226.282854] Lustre: Skipped 1 previous similar message [ 8229.343172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8229.797326] Lustre: server umount lustre-MDT0000 complete [ 8231.401322] LustreError: 262668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8231.424320] LustreError: 262668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8234.050682] LustreError: 261918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610140 with bad export cookie 15075420134228882688 [ 8234.061171] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8234.504412] Lustre: server umount lustre-MDT0001 complete [ 8246.171294] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 8249.214542] Key type lgssc unregistered [ 8249.556297] LNet: 266248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8249.567437] LNetError: 266248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8249.593227] LNet: Removed LNI 192.168.204.143@tcp [ 8250.630490] Key type .llcrypt unregistered [ 8250.636415] Key type ._llcrypt unregistered [ 8268.724114] Key type ._llcrypt registered [ 8268.729333] Key type .llcrypt registered [ 8268.898347] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 8269.979501] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8270.013049] alg: No test for adler32 (adler32-zlib) [ 8271.348458] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8271.708897] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 8273.424577] Key type lgssc registered [ 8274.706896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8283.428317] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8283.448654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8285.216424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8290.829349] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8294.370706] LustreError: 267478:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8294.393435] LustreError: 267478:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8299.497880] LustreError: 267479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8302.201622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8302.643283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8307.940937] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8317.529129] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8323.392078] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8330.142052] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8330.528633] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8337.054179] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8344.613068] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8355.077866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8365.539823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8365.542799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8365.552308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8365.565268] Lustre: Skipped 1 previous similar message [ 8365.584124] Lustre: Skipped 1 previous similar message [ 8367.611023] Lustre: server umount lustre-OST0000 complete [ 8374.240863] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8374.251117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.280237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.779715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8375.787577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8377.887368] Lustre: server umount lustre-MDT0000 complete [ 8380.904517] LustreError: 267472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8380.916346] LustreError: 267472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8381.845609] LustreError: 267459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610288 with bad export cookie 11427992679971126700 [ 8381.853518] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8381.861226] LustreError: 267459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8382.304604] Lustre: server umount lustre-MDT0001 complete [ 8393.942692] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 8397.236388] Key type lgssc unregistered [ 8397.597613] LNet: 271178:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8397.605675] LNetError: 271178:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8397.630396] LNet: Removed LNI 192.168.204.143@tcp [ 8398.592856] Key type .llcrypt unregistered [ 8398.595920] Key type ._llcrypt unregistered [ 8424.372891] Key type ._llcrypt registered [ 8424.379934] Key type .llcrypt registered [ 8424.519349] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 8425.863945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8426.053987] alg: No test for adler32 (adler32-zlib) [ 8427.154184] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8427.308636] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 8429.063816] Key type lgssc registered [ 8430.806342] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8440.543828] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8440.605591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8442.347117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8447.260456] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8452.577110] LustreError: 272591:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8452.598679] LustreError: 272591:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8456.341468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8456.993706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8461.275605] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8470.247337] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8476.712106] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8484.981078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8485.357249] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.420884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8491.346175] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8497.624033] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8523.233432] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8523.241408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8523.255241] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8527.338440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8527.338728] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8527.339921] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8529.388368] Lustre: server umount lustre-OST0000 complete [ 8533.987867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8534.000631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8534.014961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8534.026201] Lustre: Skipped 1 previous similar message [ 8537.569773] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8537.571696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8537.602793] Lustre: Skipped 1 previous similar message [ 8539.401492] Lustre: server umount lustre-MDT0000 complete [ 8542.691761] LustreError: 272585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8542.705235] LustreError: 272585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8542.828488] LustreError: 272570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610449 with bad export cookie 13007420327118353889 [ 8542.838729] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8543.162525] Lustre: server umount lustre-MDT0001 complete [ 8552.970062] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 8556.015478] Key type lgssc unregistered [ 8556.274691] LNet: 276907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8556.287202] LNetError: 276907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8556.311846] LNet: Removed LNI 192.168.204.143@tcp [ 8557.348078] Key type .llcrypt unregistered [ 8557.353747] Key type ._llcrypt unregistered [ 8574.471494] Key type ._llcrypt registered [ 8574.474534] Key type .llcrypt registered [ 8574.566934] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 8575.370043] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8575.416257] alg: No test for adler32 (adler32-zlib) [ 8576.549667] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8576.771610] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 8578.471480] Key type lgssc registered [ 8579.426740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8587.901272] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8587.942740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8589.772677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.892394] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8600.038629] LustreError: 278143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8600.069774] LustreError: 278143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8604.118987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8604.355849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8608.526756] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8615.742698] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8621.399796] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8628.008733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8628.451668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8634.390253] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8634.615218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8641.499786] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8654.083617] Lustre: server umount lustre-OST0000 complete [ 8660.458078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8660.464394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8660.474582] Lustre: Skipped 1 previous similar message [ 8660.490993] Lustre: Skipped 1 previous similar message [ 8665.568645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8665.574956] Lustre: Skipped 1 previous similar message [ 8670.689078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8670.701873] Lustre: Skipped 1 previous similar message [ 8671.711198] Lustre: 277440:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789610562/real 1789610562] req@000000007e4b90e3 x1876542601850496/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1789610578 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8671.737542] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8672.223664] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8672.367747] Lustre: server umount lustre-MDT0000 complete [ 8675.668275] LustreError: 278902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610582 with bad export cookie 14161774744931786266 [ 8675.674500] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8675.684612] LustreError: 278902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8676.096240] Lustre: server umount lustre-MDT0001 complete [ 8685.526931] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 8687.842927] Key type lgssc unregistered [ 8688.143404] LNet: 281841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8688.151255] LNetError: 281841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8689.195893] LNet: Removed LNI 192.168.204.143@tcp [ 8690.005128] Key type .llcrypt unregistered [ 8690.007121] Key type ._llcrypt unregistered [ 8698.957782] Key type ._llcrypt registered [ 8698.960264] Key type .llcrypt registered [ 8699.060415] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8714.614391] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 8727.035924] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 8727.685727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8727.795096] alg: No test for adler32 (adler32-zlib) [ 8728.826304] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 8729.041600] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 8730.719677] Key type lgssc registered [ 8731.690984] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8763.853689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8763.907695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8765.218854] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8765.261921] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8765.347550] Lustre: lustre-MDT0000: new disk, initializing [ 8765.431299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.448081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8769.793462] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8779.051238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.146565] Lustre: 285370:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8779.239494] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8779.247873] Lustre: Skipped 1 previous similar message [ 8779.326403] Lustre: lustre-MDT0001: new disk, initializing [ 8779.454662] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8779.512899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8779.526904] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8783.511752] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8791.968969] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8796.566720] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8801.860614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8802.157360] Lustre: lustre-OST0000: new disk, initializing [ 8802.161507] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8802.164890] Lustre: 287338:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8802.245949] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8807.578274] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8812.175679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8812.193190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8812.373681] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8817.243630] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8821.943169] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8822.171771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8826.484305] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8826.801489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8832.994840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8832.999649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8833.010483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8833.022251] Lustre: Skipped 1 previous similar message [ 8833.034163] LustreError: Skipped 1 previous similar message [ 8834.876738] Lustre: server umount lustre-OST0000 complete [ 8841.185646] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8841.193756] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8841.217506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8841.223204] Lustre: Skipped 1 previous similar message [ 8843.232563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8843.233541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.201496] Lustre: server umount lustre-MDT0000 complete [ 8849.020925] LustreError: 287337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610755 with bad export cookie 18255118216759472063 [ 8849.022568] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8849.039609] LustreError: 287337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8849.457213] Lustre: server umount lustre-MDT0001 complete [ 8858.095739] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 22:06:02 (1789610762) [ 8865.363262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8865.748786] LustreError: 289750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8865.774133] LustreError: 289750:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8865.846153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8869.952821] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8870.893880] LustreError: 289751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8876.006160] LustreError: 289750:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8878.484534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8878.935358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8883.457430] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8891.055703] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8896.786154] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8903.546661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8903.765556] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8910.916155] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 8919.660571] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8931.808273] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8931.819959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8931.836223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8931.844900] Lustre: Skipped 1 previous similar message [ 8933.861302] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8937.008592] Lustre: server umount lustre-OST0000 complete [ 8944.110269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8944.119475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8944.126776] Lustre: Skipped 2 previous similar messages [ 8944.150271] Lustre: Skipped 3 previous similar messages [ 8946.866634] Lustre: server umount lustre-MDT0000 complete [ 8949.216846] LustreError: 289751:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8949.241100] LustreError: 289751:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8950.657808] LustreError: 289732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789610857 with bad export cookie 18255118216759472945 [ 8950.664054] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8950.676637] LustreError: 289732:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8951.073440] Lustre: server umount lustre-MDT0001 complete [ 8963.802630] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 8967.019576] Key type lgssc unregistered [ 8967.453852] LNet: 293289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8967.462580] LNetError: 293289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8967.479530] LNet: Removed LNI 192.168.204.143@tcp [ 8968.429167] Key type .llcrypt unregistered [ 8968.432472] Key type ._llcrypt unregistered [ 8998.954598] Key type ._llcrypt registered [ 8998.964059] Key type .llcrypt registered [ 8999.128983] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 9014.422677] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 9015.357244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9015.369840] alg: No test for adler32 (adler32-zlib) [ 9016.448610] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9016.656295] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 9018.327736] Key type lgssc registered [ 9019.467947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9068.059055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9068.103320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9069.297548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9069.329371] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9069.441449] Lustre: lustre-MDT0000: new disk, initializing [ 9069.516401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9069.544638] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9074.941591] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9085.474575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9085.676594] Lustre: 297114:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9085.737647] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9085.746816] Lustre: Skipped 1 previous similar message [ 9085.844698] Lustre: lustre-MDT0001: new disk, initializing [ 9086.012256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9086.079748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9086.097596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9090.822338] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9099.753937] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9105.316916] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9112.344472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9112.587826] Lustre: lustre-OST2710: new disk, initializing [ 9112.593544] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9112.601787] Lustre: 299083:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9112.712197] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9118.716428] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9121.879097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9121.890666] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9121.993746] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9128.198859] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9135.381505] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9135.567821] Lustre: lustre-OST03e8: new disk, initializing [ 9135.572284] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9135.579435] Lustre: 300146:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9135.675545] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9142.650542] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9143.477629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9143.493069] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9143.612459] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9151.260814] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9160.869852] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9161.254311] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9166.420355] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9166.733160] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9171.596964] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9171.834661] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9176.665540] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9176.925446] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9183.201249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9183.214883] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9183.246285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9184.029885] Lustre: server umount lustre-MDT0000 complete [ 9184.751891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9184.770350] Lustre: Skipped 1 previous similar message [ 9187.750907] LustreError: 297105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611094 with bad export cookie 1317243049016844311 [ 9187.761861] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9188.110712] Lustre: server umount lustre-MDT0001 complete [ 9201.980797] Lustre: server umount lustre-OST2710 complete [ 9215.927201] Lustre: server umount lustre-OST03e8 complete [ 9236.960858] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [ 9249.848891] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 9288.854144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9289.120441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9289.155443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9289.227823] Lustre: lustre-MDT0000: new disk, initializing [ 9289.305720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.324715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9293.941950] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9305.900793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9306.041928] Lustre: 305397:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9306.112371] LustreError: 305409:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9306.142456] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9306.144736] LustreError: 305409:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9306.155285] Lustre: Skipped 1 previous similar message [ 9306.213789] Lustre: lustre-MDT0001: new disk, initializing [ 9306.290471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9306.340357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9306.356095] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9311.113808] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9320.902719] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9326.645959] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9334.709122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9334.974341] Lustre: lustre-OST0000: new disk, initializing [ 9334.977568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9334.983731] Lustre: 307365:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9335.061321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9341.156546] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9344.058828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9344.076139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9344.191484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9351.123611] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9356.027652] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9356.238481] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9361.118861] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9361.318437] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9364.974696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9364.984196] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.992770] Lustre: Skipped 1 previous similar message [ 9364.998066] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9365.001451] Lustre: Skipped 1 previous similar message [ 9369.353542] Lustre: server umount lustre-OST0000 complete [ 9375.200483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9375.205982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9375.216153] Lustre: Skipped 2 previous similar messages [ 9375.249617] Lustre: Skipped 1 previous similar message [ 9379.825600] Lustre: server umount lustre-MDT0000 complete [ 9380.320937] LustreError: 305405:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9380.361397] LustreError: 305405:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9383.946124] LustreError: 306193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611290 with bad export cookie 1317243049016845991 [ 9383.956401] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9383.971462] LustreError: 306193:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9384.463609] Lustre: server umount lustre-MDT0001 complete [ 9394.616890] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 22:14:58 (1789611298) [ 9396.430059] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9398.374192] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 22:15:03 (1789611303) [ 9401.928544] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 9428.211096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9428.372391] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9428.386374] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9428.498119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9433.053670] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9440.792530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9440.873572] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9440.894980] Lustre: 311222:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9445.640138] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9466.616695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9466.741621] Lustre: Denying initial registration attempt from nid 192.168.204.143@tcp, specified as failover [ 9466.749643] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9466.756125] LustreError: 312676:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 9466.761336] LustreError: 312676:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 9466.770395] LustreError: 312676:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9466.882363] Lustre: server umount lustre-OST0000 complete [ 9466.885927] LustreError: 312676:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9475.555377] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9475.559188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9475.566193] Lustre: Skipped 1 previous similar message [ 9480.427737] Lustre: server umount lustre-MDT0000 complete [ 9480.675768] LustreError: 311234:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9480.698768] LustreError: 311234:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9484.012408] LustreError: 311214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611390 with bad export cookie 1317243049016846866 [ 9484.016581] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9484.024074] LustreError: 311214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9484.326762] Lustre: server umount lustre-MDT0001 complete [ 9496.229253] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 9499.445068] Key type lgssc unregistered [ 9499.767708] LNet: 313796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9499.781371] LNetError: 313796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9500.842081] LNet: Removed LNI 192.168.204.143@tcp [ 9502.051956] Key type .llcrypt unregistered [ 9502.057504] Key type ._llcrypt unregistered [ 9514.449430] Key type ._llcrypt registered [ 9514.453115] Key type .llcrypt registered [ 9514.597707] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 9515.758033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9515.851099] alg: No test for adler32 (adler32-zlib) [ 9517.051887] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9517.429447] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 9519.087619] Key type lgssc registered [ 9520.318880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9566.955414] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 9575.893839] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9575.913694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9577.236723] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9577.264288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9577.593860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9582.296350] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9591.740275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.804503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9591.830073] Lustre: 316559:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9592.028724] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9595.967752] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9604.023959] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9610.299358] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9629.996375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9630.229765] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9630.520858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9636.746858] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9645.267918] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9649.636118] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9649.650963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9649.680130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9650.145681] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9650.161891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9653.527299] Lustre: server umount lustre-OST0000 complete [ 9658.855488] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9658.863486] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9658.875125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9658.884584] Lustre: Skipped 2 previous similar messages [ 9660.384607] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9660.389663] Lustre: Skipped 1 previous similar message [ 9663.441763] Lustre: server umount lustre-MDT0000 complete [ 9667.504435] LustreError: 316551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611574 with bad export cookie 17784023777800063370 [ 9667.512289] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9667.520995] LustreError: 316551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9667.797901] Lustre: server umount lustre-MDT0001 complete [ 9679.090708] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 9682.559430] Key type lgssc unregistered [ 9682.817114] LNet: 320132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9682.827319] LNetError: 320132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9682.843832] LNet: Removed LNI 192.168.204.143@tcp [ 9684.143539] Key type .llcrypt unregistered [ 9684.147712] Key type ._llcrypt unregistered [ 9714.214254] Key type ._llcrypt registered [ 9714.225234] Key type .llcrypt registered [ 9714.353953] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 9715.317691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9715.432474] alg: No test for adler32 (adler32-zlib) [ 9716.541502] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9716.809289] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 9718.543506] Key type lgssc registered [ 9719.830870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9727.628430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9727.670845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9729.124710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9733.935767] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9739.237437] LustreError: 321547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9739.268564] LustreError: 321547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9742.572433] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9743.073823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9747.192180] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9754.614403] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9761.101702] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9770.094255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9770.631084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9777.733326] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9786.670485] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9798.627328] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9798.644900] Lustre: Skipped 1 previous similar message [ 9798.653413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9798.660325] Lustre: Skipped 2 previous similar messages [ 9799.506755] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.43@tcp (stopping) [ 9801.344508] Lustre: server umount lustre-MDT0000 complete [ 9803.751579] LustreError: 321543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9803.793674] LustreError: 321543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9805.694538] LustreError: 321527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611712 with bad export cookie 13809197999673913551 [ 9805.704586] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9805.715546] LustreError: 321527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9808.866509] LustreError: 321544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9808.871904] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.905022] LustreError: 321544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9808.951733] Lustre: Skipped 1 previous similar message [ 9808.963179] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9810.929299] LustreError: 321544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9810.963725] LustreError: 321544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9814.848714] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.43@tcp (stopping) [ 9814.861687] Lustre: Skipped 2 previous similar messages [ 9816.032241] LustreError: 321544:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9816.069533] LustreError: 321544:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9819.970068] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.43@tcp (stopping) [ 9819.988977] Lustre: Skipped 1 previous similar message [ 9820.129069] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9820.932921] Lustre: server umount lustre-MDT0001 complete [ 9829.429451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9839.017574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9851.390780] LustreError: 323380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9851.416489] LustreError: 323380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9851.514843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9851.585644] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9856.659197] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9865.720430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9866.090780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9870.645920] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9871.214202] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9879.152420] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9880.881630] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9887.700952] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9889.523134] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9896.932086] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9896.940609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9896.945549] Lustre: Skipped 1 previous similar message [ 9896.946430] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9896.957982] Lustre: Skipped 1 previous similar message [ 9909.216563] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9909.452839] Lustre: server umount lustre-OST0000 complete [ 9917.408626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9917.415307] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9917.434430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9917.439467] Lustre: Skipped 1 previous similar message [ 9917.450781] Lustre: Skipped 5 previous similar messages [ 9920.014722] Lustre: server umount lustre-MDT0000 complete [ 9922.535880] LustreError: 325147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9922.561193] LustreError: 325147:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9924.365794] LustreError: 321527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611831 with bad export cookie 13809197999673921251 [ 9924.369458] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9924.389695] LustreError: 321527:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9924.947291] Lustre: server umount lustre-MDT0001 complete [ 9936.430095] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [ 9939.993936] Key type lgssc unregistered [ 9940.325086] LNet: 327820:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9940.344364] LNetError: 327820:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9940.370118] LNet: Removed LNI 192.168.204.143@tcp [ 9941.497453] Key type .llcrypt unregistered [ 9941.500057] Key type ._llcrypt unregistered [ 9964.247193] Key type ._llcrypt registered [ 9964.249022] Key type .llcrypt registered [ 9964.397813] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [ 9965.844966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9965.888021] alg: No test for adler32 (adler32-zlib) [ 9966.959485] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [ 9967.262417] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [ 9969.007249] Key type lgssc registered [ 9970.349531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9978.231710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9978.258926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.686701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.041500] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [ 9990.112876] LustreError: 329232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.149611] LustreError: 329232:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9993.200746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9993.662452] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9997.664742] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10005.200348] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10011.097396] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10014.176883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10014.181399] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10014.207546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10015.207040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10015.210726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10018.343574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10018.361125] Lustre: Skipped 2 previous similar messages [10020.491087] Lustre: server umount lustre-MDT0000 complete [10023.393092] LustreError: 329961:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.416510] LustreError: 329961:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10024.513181] LustreError: 329214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789611931 with bad export cookie 4093044450213191049 [10024.518944] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10024.532402] LustreError: 329214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10025.216571] Lustre: server umount lustre-MDT0001 complete [10045.622146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10046.053362] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10046.083640] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10046.415621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10051.380965] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10062.814702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10062.948764] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10062.987917] Lustre: 331821:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10063.502519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10069.349818] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10077.141120] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10083.914645] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10094.320092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10094.590282] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10094.602396] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10094.612315] LustreError: 333707:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10094.626056] LustreError: 333707:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10094.631272] LustreError: 333707:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10094.719364] Lustre: server umount lustre-OST0000 complete [10094.723141] LustreError: 333707:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10105.707625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10105.958432] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10106.278752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10113.175815] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10116.244942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10120.528376] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10128.631953] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10128.734538] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10128.807547] Lustre: lustre-OST0001: new disk, initializing [10128.812673] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10128.818038] Lustre: 335251:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10128.913392] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10135.643242] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10139.316507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10139.334879] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10139.503836] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10144.352759] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10149.866170] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10149.871273] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10149.902252] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10149.910772] Lustre: Skipped 1 previous similar message [10153.707186] Lustre: server umount lustre-OST0001 complete [10154.991622] LustreError: 335772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10155.027528] LustreError: 335772:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10160.100084] LustreError: 334174:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10160.114135] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10160.127576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10160.139984] LustreError: 334174:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10160.173401] Lustre: Skipped 2 previous similar messages [10164.668880] Lustre: server umount lustre-OST0000 complete [10170.342147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10170.364449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10170.375918] Lustre: Skipped 2 previous similar messages [10175.251346] Lustre: server umount lustre-MDT0000 complete [10175.466549] LustreError: 331829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10175.501022] LustreError: 331829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10179.919328] LustreError: 331814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789612086 with bad export cookie 4093044450213191602 [10179.919933] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10179.948876] LustreError: 331814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10180.652242] Lustre: server umount lustre-MDT0001 complete [10193.811432] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [10197.043797] Key type lgssc unregistered [10197.459986] LNet: 337208:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10197.471105] LNetError: 337208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10197.510210] LNet: Removed LNI 192.168.204.143@tcp [10198.441193] Key type .llcrypt unregistered [10198.445348] Key type ._llcrypt unregistered [10268.629139] Key type ._llcrypt registered [10268.630859] Key type .llcrypt registered [10268.719401] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [10281.248193] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [10282.381515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10282.436122] alg: No test for adler32 (adler32-zlib) [10283.505889] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [10283.809596] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [10285.519278] Key type lgssc registered [10286.560661] Lustre: Echo OBD driver; http://www.lustre.org/ [10323.026718] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10323.103523] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.516990] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10324.555539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10324.700603] Lustre: lustre-MDT0000: new disk, initializing [10324.830182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10324.849624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10329.787173] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10342.833435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10342.990630] Lustre: 342118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10343.105417] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10343.121331] Lustre: Skipped 1 previous similar message [10343.216709] Lustre: lustre-MDT0001: new disk, initializing [10343.347397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10343.417381] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10343.434352] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10348.411931] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10359.094434] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10365.742279] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10373.760098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10374.054539] Lustre: lustre-OST0000: new disk, initializing [10374.057707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10374.066980] Lustre: 344088:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10374.164713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10381.310287] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10382.939561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10382.950586] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10383.067070] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10390.732855] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10395.176718] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10395.443647] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10400.079308] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10400.273516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10403.812401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10403.820463] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10403.842091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10403.847699] Lustre: Skipped 1 previous similar message [10408.253281] Lustre: server umount lustre-OST0000 complete [10414.048494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10414.069863] Lustre: Skipped 1 previous similar message [10414.075626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10414.081187] Lustre: Skipped 1 previous similar message [10419.169502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10419.179502] Lustre: Skipped 1 previous similar message [10423.265165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10423.268506] Lustre: Skipped 1 previous similar message [10426.335154] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10426.577936] Lustre: server umount lustre-MDT0000 complete [10430.215671] LustreError: 342109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789612336 with bad export cookie 3985833734893679907 [10430.225042] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10430.235203] LustreError: 342109:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10430.625319] Lustre: server umount lustre-MDT0001 complete [10439.108243] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:32:24 (1789612344) [10448.199027] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:32:32 (1789612352) [10461.115632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10461.724449] LustreError: 346847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10461.744113] LustreError: 346847:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10461.877792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10467.311725] LustreError: 346848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10469.658908] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10471.393402] LustreError: 346847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10476.515530] LustreError: 346848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10481.638395] LustreError: 346847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10481.729083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10482.538035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10489.221150] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10498.004331] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10503.966598] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10511.220740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10511.654034] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10518.788987] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10527.867321] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10541.146612] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10543.234985] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10545.008377] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10546.878677] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10548.806808] Lustre: Failing over lustre-MDT0000 [10549.064577] Lustre: server umount lustre-MDT0000 complete [10549.218453] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10549.224120] LustreError: Skipped 1 previous similar message [10549.226774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10549.236172] Lustre: Skipped 1 previous similar message [10549.242322] LustreError: 346847:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.803508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10565.430477] LustreError: 346843:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10565.448629] LustreError: 346843:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [10568.132579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10568.255805] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10568.499479] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10568.581211] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10571.577243] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10573.684531] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10573.818287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10573.861450] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [10573.937869] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10583.101988] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10584.765902] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10588.126457] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10590.271672] Lustre: Failing over lustre-MDT0000 [10590.523941] Lustre: server umount lustre-MDT0000 complete [10594.275209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.301215] Lustre: Skipped 3 previous similar messages [10599.395705] LustreError: 346844:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10599.420196] LustreError: 346844:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [10610.260362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.404549] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10610.816875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10610.896837] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10612.036977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10616.311818] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10616.323049] Lustre: Skipped 2 previous similar messages [10616.370902] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10616.444240] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10616.465669] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10627.105135] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10628.766271] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10636.771717] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10636.794214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10636.801899] Lustre: Skipped 2 previous similar messages [10636.802295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10636.830815] Lustre: Skipped 2 previous similar messages [10641.184640] Lustre: server umount lustre-MDT0000 complete [10644.811240] LustreError: 346829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789612551 with bad export cookie 3985833734893682616 [10644.824211] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10644.827124] LustreError: 346829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10645.209663] Lustre: server umount lustre-MDT0001 complete [10659.631919] Lustre: server umount lustre-OST0000 complete [10678.371493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10678.995320] LustreError: 353209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10679.031573] LustreError: 353209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [10679.109687] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10684.343647] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10693.572589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10698.415924] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10708.362960] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10715.421449] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10722.611231] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10724.029888] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10728.928208] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10736.732537] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10748.897368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10748.900577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10748.908201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10748.920392] Lustre: Skipped 1 previous similar message [10748.935769] Lustre: Skipped 1 previous similar message [10752.344978] Lustre: server umount lustre-OST0000 complete [10768.356188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10768.360922] Lustre: Skipped 5 previous similar messages [10770.399144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10770.606427] Lustre: server umount lustre-MDT0000 complete [10775.042719] LustreError: 353189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789612681 with bad export cookie 3985833734893683365 [10775.049180] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10775.050781] LustreError: 353189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10775.534363] Lustre: server umount lustre-MDT0001 complete [10787.919453] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [10791.018632] Key type lgssc unregistered [10791.259782] LNet: 356744:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10791.268622] LNetError: 356744:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10791.295151] LNet: Removed LNI 192.168.204.143@tcp [10792.777829] Key type .llcrypt unregistered [10792.782317] Key type ._llcrypt unregistered [10830.330468] Key type ._llcrypt registered [10830.331683] Key type .llcrypt registered [10830.437287] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [10831.340888] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10831.428844] alg: No test for adler32 (adler32-zlib) [10832.736578] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [10833.208232] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [10835.023785] Key type lgssc registered [10836.107615] Lustre: Echo OBD driver; http://www.lustre.org/ [10844.935440] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10844.978054] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10846.348180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10851.897584] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10856.935059] LustreError: 358257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10856.962279] LustreError: 358257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10861.386820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10861.899955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10866.662099] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10874.406946] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10880.836845] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10888.583908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10889.087242] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10892.156922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10896.559421] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10906.046581] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10918.368621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10918.376591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10918.391312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10919.690718] Lustre: server umount lustre-MDT0000 complete [10921.952909] LustreError: 358257:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10921.963980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10923.622315] LustreError: 358237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789612830 with bad export cookie 13749521713549872599 [10923.625325] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10923.632904] LustreError: 358237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10924.119229] Lustre: server umount lustre-MDT0001 complete [10939.140573] Lustre: server umount lustre-OST0000 complete [10956.652596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10957.523019] LustreError: 361968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.543285] LustreError: 361968:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10957.713155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10962.913373] LustreError: 361969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10963.521838] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10968.034027] LustreError: 361968:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10973.072212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10973.658045] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10978.973381] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [10987.577206] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10994.274814] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11001.406338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11001.781852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11002.833192] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11008.890531] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11015.854279] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11029.473164] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11029.476398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11029.485935] Lustre: Skipped 3 previous similar messages [11029.489551] Lustre: Skipped 1 previous similar message [11033.018555] Lustre: server umount lustre-MDT0000 complete [11034.593990] LustreError: 361969:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11034.620030] LustreError: 361969:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11036.936772] LustreError: 361950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789612943 with bad export cookie 13749521713549873719 [11036.951575] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11037.229709] Lustre: server umount lustre-MDT0001 complete [11051.858342] Lustre: server umount lustre-OST0000 complete [11070.776981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11071.717980] LustreError: 365880:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11071.823539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11078.316835] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11089.793558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11095.300496] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11103.621307] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11110.370775] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11117.666710] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11117.965644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11117.974473] Lustre: Skipped 1 previous similar message [11124.829264] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11132.656560] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11141.500703] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11149.793783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11149.803367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11149.827224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11149.834245] Lustre: Skipped 1 previous similar message [11151.850168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11154.702767] Lustre: server umount lustre-OST0000 complete [11162.086133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11162.099149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11162.109199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11162.113567] Lustre: Skipped 2 previous similar messages [11162.139064] Lustre: Skipped 2 previous similar messages [11164.881830] Lustre: server umount lustre-MDT0000 complete [11167.200289] LustreError: 367206:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11167.219032] LustreError: 367206:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11168.756488] LustreError: 365861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613075 with bad export cookie 13749521713549874741 [11168.765081] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11168.781341] LustreError: 365861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11169.332247] Lustre: server umount lustre-MDT0001 complete [11182.413691] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [11186.158070] Key type lgssc unregistered [11186.534564] LNet: 369435:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11186.547914] LNetError: 369435:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11186.565969] LNet: Removed LNI 192.168.204.143@tcp [11187.753831] Key type .llcrypt unregistered [11187.759654] Key type ._llcrypt unregistered [11216.426748] Key type ._llcrypt registered [11216.429569] Key type .llcrypt registered [11216.521967] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [11217.603114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11217.625018] alg: No test for adler32 (adler32-zlib) [11218.926373] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [11219.134205] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [11220.839448] Key type lgssc registered [11222.093094] Lustre: Echo OBD driver; http://www.lustre.org/ [11231.452862] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11231.474318] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11231.489491] LustreError: 370873:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11231.530640] LustreError: 370873:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11231.544578] LustreError: 370873:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11231.555656] LustreError: 370873:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11231.565738] LustreError: 370873:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11240.153985] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11240.159470] LustreError: 371270:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11240.184483] LustreError: 371270:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11240.198350] LustreError: 371270:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11240.204570] LustreError: 371270:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11240.215363] LustreError: 371270:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11263.303996] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [11267.276794] Key type lgssc unregistered [11267.795288] LNet: 372273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11267.802590] LNetError: 372273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11267.841126] LNet: Removed LNI 192.168.204.143@tcp [11269.088091] Key type .llcrypt unregistered [11269.090513] Key type ._llcrypt unregistered [11291.687213] Key type ._llcrypt registered [11291.689415] Key type .llcrypt registered [11291.821134] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [11307.712914] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [11308.804621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11309.003045] alg: No test for adler32 (adler32-zlib) [11310.274717] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [11310.628679] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [11312.367284] Key type lgssc registered [11313.782690] Lustre: Echo OBD driver; http://www.lustre.org/ [11349.250890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11349.356395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11350.836745] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11350.891905] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11350.988609] Lustre: lustre-MDT0000: new disk, initializing [11351.094907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11351.120933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11355.742159] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11366.567252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11366.684857] Lustre: 375430:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11366.744310] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11366.747747] Lustre: Skipped 1 previous similar message [11366.869401] Lustre: lustre-MDT0001: new disk, initializing [11366.976637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11367.040237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11367.056378] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11371.605280] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11381.038910] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11386.537160] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11392.670955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11392.919476] Lustre: lustre-OST0000: new disk, initializing [11392.923747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11392.930249] Lustre: 377400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11393.028978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11398.845500] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11402.331968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11402.360324] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11402.489922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11408.207508] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11413.239524] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11413.442296] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11418.541538] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11418.726338] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11423.205910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11423.210952] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11423.214749] LustreError: Skipped 1 previous similar message [11423.216046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11423.241130] Lustre: Skipped 1 previous similar message [11423.264221] Lustre: Skipped 1 previous similar message [11426.946965] Lustre: server umount lustre-OST0000 complete [11433.442432] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11433.447223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11433.450318] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11433.463886] Lustre: Skipped 1 previous similar message [11438.563924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11438.576959] Lustre: Skipped 2 previous similar messages [11443.684806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11443.698033] Lustre: Skipped 1 previous similar message [11446.239433] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11446.461863] Lustre: server umount lustre-MDT0000 complete [11450.119555] LustreError: 377399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613356 with bad export cookie 4448467727359257972 [11450.124786] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11450.134441] LustreError: 377399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11450.519982] Lustre: server umount lustre-MDT0001 complete [11459.962589] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11461.673666] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:49:26 (1789613366) [11468.435319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11469.021780] LustreError: 379860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11469.043471] LustreError: 379860:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11469.099932] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11474.129333] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11474.404449] LustreError: 379861:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11478.499113] LustreError: 379860:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11483.054491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11483.518900] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11487.821770] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11494.639517] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11501.916755] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11508.403492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11508.713294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11515.723659] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11523.104839] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11529.912525] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11530.098856] Lustre: lustre-OST0001: new disk, initializing [11530.105418] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11530.111844] Lustre: 382694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11530.218843] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11535.944437] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11535.955522] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11536.076781] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11536.264435] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11546.320436] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11551.716990] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.729159] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11551.733519] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11551.735026] LustreError: Skipped 1 previous similar message [11551.765555] Lustre: Skipped 1 previous similar message [11557.311971] Lustre: server umount lustre-OST0001 complete [11559.218377] LustreError: 383028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.078898] LustreError: 381681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11567.100213] LustreError: 381681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11575.265899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11575.274431] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11575.286391] Lustre: Skipped 1 previous similar message [11575.296662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11575.305904] Lustre: Skipped 3 previous similar messages [11577.316327] LustreError: 381679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11577.332973] LustreError: 381679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11580.418753] Lustre: server umount lustre-OST0000 complete [11586.016986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11586.028240] LustreError: Skipped 1 previous similar message [11586.033403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11586.044515] Lustre: Skipped 1 previous similar message [11590.667921] Lustre: server umount lustre-MDT0000 complete [11594.375765] LustreError: 379842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613501 with bad export cookie 4448467727359258854 [11594.393914] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11600.798732] Lustre: server umount lustre-MDT0001 complete [11612.284076] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [11615.120721] Key type lgssc unregistered [11615.409624] LNet: 384684:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11615.422253] LNetError: 384684:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11615.434544] LNet: Removed LNI 192.168.204.143@tcp [11616.315877] Key type .llcrypt unregistered [11616.321575] Key type ._llcrypt unregistered [11671.350080] Key type ._llcrypt registered [11671.353437] Key type .llcrypt registered [11671.568721] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [11672.945219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11673.254976] alg: No test for adler32 (adler32-zlib) [11674.484400] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [11674.807067] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [11676.703254] Key type lgssc registered [11678.349789] Lustre: Echo OBD driver; http://www.lustre.org/ [11687.712626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11687.738511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11689.110595] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11689.132363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11689.327192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11694.217916] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11703.709463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11703.786353] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11703.817604] Lustre: 387481:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11704.209745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11709.613281] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11718.316610] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11724.749701] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11728.865782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11728.873144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11728.895829] Lustre: Skipped 1 previous similar message [11733.409076] Lustre: server umount lustre-MDT0000 complete [11738.385982] LustreError: 387474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613645 with bad export cookie 11182589217828525615 [11738.398199] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11738.407300] LustreError: 387474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11738.782079] Lustre: server umount lustre-MDT0001 complete [11749.620607] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:54:14 (1789613654) [11756.568928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.290522] LustreError: 389994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11757.319143] LustreError: 389994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11757.432917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11762.667844] LustreError: 389995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.828295] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11767.776136] LustreError: 389994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11772.675557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11772.905953] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [11772.916564] Lustre: Skipped 1 previous similar message [11773.223992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11777.523566] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11786.100198] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11792.062953] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11798.849472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11798.989450] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11799.164809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11805.616852] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11813.447130] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11820.581634] Lustre: Permanently deactivating lustre-OST0000 [11820.598414] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11830.353991] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11832.333978] LustreError: 393029:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11832.353640] LustreError: 393029:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11832.376844] LustreError: 393029:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11835.886409] LustreError: 393078:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11835.896725] LustreError: 393078:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11844.242141] Lustre: server umount lustre-OST0000 complete [11847.185073] LustreError: 393328:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11847.207475] LustreError: 393328:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11850.218101] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11850.228571] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11850.244375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11852.257000] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11852.260468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11852.289497] Lustre: Skipped 1 previous similar message [11857.383061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11857.392960] Lustre: Skipped 1 previous similar message [11858.523219] Lustre: server umount lustre-MDT0000 complete [11862.498688] LustreError: 389991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11862.513585] LustreError: 389991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11863.029162] LustreError: 389974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613769 with bad export cookie 11182589217828526196 [11863.040537] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11863.049297] LustreError: 389974:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11872.454534] Lustre: server umount lustre-MDT0001 complete [11882.705760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11887.507824] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11890.377541] LustreError: 394489:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11896.498913] Lustre: 394827:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.143@tcp [11898.531954] Lustre: server umount MGS complete [11909.844792] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11910.352525] LustreError: 395551:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11910.378092] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11910.383061] Lustre: Skipped 1 previous similar message [11910.419143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11914.914993] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11920.867475] LustreError: 395552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11920.890554] LustreError: 395552:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11923.693664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11924.093795] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11929.172990] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [11937.704284] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11944.506598] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11946.382945] Lustre: Permanently reactivating lustre-OST0000 [11946.404656] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11946.413580] Lustre: Skipped 2 previous similar messages [11976.161303] LustreError: 397384:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11985.209229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11985.673567] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11985.680849] Lustre: Skipped 1 previous similar message [11993.808383] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12002.259029] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12007.399058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.408918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12007.419044] Lustre: Skipped 1 previous similar message [12007.432434] Lustre: Skipped 1 previous similar message [12009.180335] Lustre: server umount lustre-OST0000 complete [12016.615178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12016.623445] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12020.050610] Lustre: server umount lustre-MDT0000 complete [12022.752256] LustreError: 395546:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12022.774647] LustreError: 395546:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12024.092182] LustreError: 395532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789613930 with bad export cookie 11182589217828527246 [12024.094210] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12024.107167] LustreError: 395532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12024.555358] Lustre: server umount lustre-MDT0001 complete [12037.941601] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [12042.327470] Key type lgssc unregistered [12042.697686] LNet: 399966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12042.711979] LNetError: 399966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12042.732196] LNet: Removed LNI 192.168.204.143@tcp [12044.023617] Key type .llcrypt unregistered [12044.028444] Key type ._llcrypt unregistered [12066.420951] Key type ._llcrypt registered [12066.422739] Key type .llcrypt registered [12066.538960] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [12081.092218] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [12082.129369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12082.153220] alg: No test for adler32 (adler32-zlib) [12083.300576] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [12083.652191] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [12085.392155] Key type lgssc registered [12086.662407] Lustre: Echo OBD driver; http://www.lustre.org/ [12129.292363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12129.362363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12130.659288] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12130.697574] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12130.821450] Lustre: lustre-MDT0000: new disk, initializing [12130.950933] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12130.997829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12135.591580] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12148.252034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12148.512572] Lustre: 403120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12148.634200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12148.638872] Lustre: Skipped 1 previous similar message [12148.742132] Lustre: lustre-MDT0001: new disk, initializing [12148.907894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12148.975213] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12149.001781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12156.105864] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12166.661282] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12173.687557] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12181.667622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12182.062649] Lustre: lustre-OST0000: new disk, initializing [12182.069251] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12182.078815] Lustre: 405099:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12182.278270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12187.740561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12187.765976] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12187.878133] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12189.175720] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12198.597971] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12203.748456] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12203.916124] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12209.323989] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12209.553594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12213.733245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12213.733561] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12213.749233] LustreError: Skipped 1 previous similar message [12213.754431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12213.775912] Lustre: Skipped 1 previous similar message [12213.796093] Lustre: Skipped 1 previous similar message [12217.720458] Lustre: server umount lustre-OST0000 complete [12223.974794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12223.988271] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12223.997123] Lustre: Skipped 1 previous similar message [12224.036673] Lustre: Skipped 1 previous similar message [12228.125601] Lustre: server umount lustre-MDT0000 complete [12231.917960] LustreError: 403114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614138 with bad export cookie 2553529450329408246 [12231.919405] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12231.942773] LustreError: 403114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12232.256127] Lustre: server umount lustre-MDT0001 complete [12242.529122] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12245.027911] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 23:02:29 (1789614149) [12253.296284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.884481] LustreError: 407562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12253.912774] LustreError: 407562:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12254.005584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12259.288669] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12259.298740] LustreError: 407563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12263.395798] LustreError: 407562:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.523153] LustreError: 407563:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12268.729575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.169367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12274.343690] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12281.818504] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12287.670377] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12295.867967] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12296.263863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12304.069436] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12314.178454] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12316.827221] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12319.654900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12328.931455] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12328.935618] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12328.999171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12332.692803] Lustre: server umount lustre-OST0000 complete [12339.188919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12339.196230] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12339.211576] Lustre: Skipped 2 previous similar messages [12339.239957] Lustre: Skipped 2 previous similar messages [12344.078759] Lustre: server umount lustre-MDT0000 complete [12344.291369] LustreError: 408291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12344.326615] LustreError: 408291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12349.369503] LustreError: 407544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614256 with bad export cookie 2553529450329409128 [12349.390307] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12349.789224] Lustre: server umount lustre-MDT0001 complete [12361.269555] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [12364.548276] Key type lgssc unregistered [12364.959529] LNet: 411289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12364.969858] LNetError: 411289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12365.001663] LNet: Removed LNI 192.168.204.143@tcp [12366.074492] Key type .llcrypt unregistered [12366.077595] Key type ._llcrypt unregistered [12396.181910] Key type ._llcrypt registered [12396.185851] Key type .llcrypt registered [12396.310897] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [12396.631030] Key type .llcrypt unregistered [12396.635984] Key type ._llcrypt unregistered [12416.972336] Key type ._llcrypt registered [12416.974910] Key type .llcrypt registered [12417.096853] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [12418.120603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.176299] alg: No test for adler32 (adler32-zlib) [12419.234684] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [12419.467900] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [12421.095225] Key type lgssc registered [12422.639566] Lustre: Echo OBD driver; http://www.lustre.org/ [12432.820482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12432.868058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12434.386610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12439.551763] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12444.643212] LustreError: 413497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12444.660589] LustreError: 413497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12448.729926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12449.051186] LustreError: 413496:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12449.072600] LustreError: 413496:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12449.088863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12454.379444] LustreError: 413497:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12454.400469] LustreError: 413497:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12455.662516] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12459.504650] LustreError: 413492:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12459.537995] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12459.559956] LustreError: 413492:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12464.407744] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12464.617370] LustreError: 413493:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12464.658710] LustreError: 413493:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12471.692907] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12472.085939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12476.593683] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12485.092673] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12485.108731] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12485.133959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12486.987149] Lustre: server umount lustre-OST0000 complete [12492.770538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12492.771796] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12492.782077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12492.796323] Lustre: Skipped 1 previous similar message [12492.832131] LustreError: Skipped 1 previous similar message [12493.428935] Lustre: server umount lustre-MDT0000 complete [12497.216301] LustreError: 413477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614403 with bad export cookie 10243577231912315506 [12497.219667] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12497.222604] LustreError: 413477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12507.817070] Lustre: server umount lustre-MDT0001 complete [12521.877601] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [12525.177398] Key type lgssc unregistered [12525.489809] LNet: 416643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12525.519317] LNetError: 416643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12525.566248] LNet: Removed LNI 192.168.204.143@tcp [12527.379403] Key type .llcrypt unregistered [12527.381674] Key type ._llcrypt unregistered [12556.192739] Key type ._llcrypt registered [12556.195753] Key type .llcrypt registered [12556.283161] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [12557.516234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12557.618143] alg: No test for adler32 (adler32-zlib) [12558.684572] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [12558.923027] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [12560.599259] Key type lgssc registered [12561.756865] Lustre: Echo OBD driver; http://www.lustre.org/ [12570.639219] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12570.666447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12586.272664] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12591.455915] LustreError: 418032:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.143@tcp: failed processing log, type 4: rc = -110 [12618.143239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12624.713448] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12632.328273] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12639.515809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12640.053769] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12640.161532] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12645.032740] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12645.354315] LustreError: 418058:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12645.384313] LustreError: 418058:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12650.469249] LustreError: 418057:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.517260] LustreError: 419067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12652.552565] LustreError: 419067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12653.999783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.423811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12659.281607] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12667.114907] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12673.598463] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12680.164513] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12680.195950] Lustre: Skipped 1 previous similar message [12680.198600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12680.207309] Lustre: Skipped 1 previous similar message [12682.540806] Lustre: server umount lustre-OST0000 complete [12687.855754] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12687.868208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12687.891336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12689.624741] Lustre: server umount lustre-MDT0000 complete [12690.401137] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12690.401905] LustreError: 419074:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12690.428966] LustreError: 419074:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12694.704962] LustreError: 418039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614601 with bad export cookie 3258971624103025402 [12694.718776] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12694.727185] LustreError: 418039:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12695.191702] Lustre: server umount lustre-MDT0001 complete [12709.441599] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [12712.778612] Key type lgssc unregistered [12713.119261] LNet: 421601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12713.125807] LNetError: 421601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12713.147864] LNet: Removed LNI 192.168.204.143@tcp [12714.479288] Key type .llcrypt unregistered [12714.485502] Key type ._llcrypt unregistered [12744.545672] Key type ._llcrypt registered [12744.548681] Key type .llcrypt registered [12744.693075] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [12746.168386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12746.209530] alg: No test for adler32 (adler32-zlib) [12747.356164] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [12747.573084] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [12749.239264] Key type lgssc registered [12750.405984] Lustre: Echo OBD driver; http://www.lustre.org/ [12758.307086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12758.359058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12759.948175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12764.820764] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12769.257895] LustreError: 423015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12769.291475] LustreError: 423015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12774.383532] LustreError: 423014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12777.086637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12777.471351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12783.502063] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12792.415677] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12798.924417] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12807.472025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12807.841612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12815.936790] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12824.712587] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12829.391632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12834.788318] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12834.791247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12834.819154] Lustre: Skipped 1 previous similar message [12834.831698] Lustre: Skipped 2 previous similar messages [12836.188997] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.43@tcp (stopping) [12839.908091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12839.929513] Lustre: Skipped 2 previous similar messages [12840.194517] Lustre: server umount lustre-MDT0000 complete [12841.275961] LustreError: 424149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.43@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12842.994905] LustreError: 422994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.43@tcp arrived at 1789614749 with bad export cookie 5094473836954144974 [12843.012038] LustreError: 422994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12845.026201] LustreError: 423015:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12845.045903] LustreError: 423015:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12845.107366] LustreError: 422996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789614751 with bad export cookie 5094473836954144302 [12845.119636] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12849.120079] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12849.134568] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12849.145904] Lustre: Skipped 1 previous similar message [12849.151064] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12850.144723] LustreError: 423014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12850.176727] LustreError: 423014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12851.318962] Lustre: server umount lustre-OST0000 complete [12858.218693] Lustre: server umount lustre-MDT0001 complete [12868.866654] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [12871.746555] Key type lgssc unregistered [12872.253679] LNet: 426701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12872.268453] LNetError: 426701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12872.297938] LNet: Removed LNI 192.168.204.143@tcp [12873.593648] Key type .llcrypt unregistered [12873.597270] Key type ._llcrypt unregistered [12901.912319] Key type ._llcrypt registered [12901.913991] Key type .llcrypt registered [12902.049618] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [12903.567515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12903.785688] alg: No test for adler32 (adler32-zlib) [12905.214622] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [12905.511636] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [12907.242435] Key type lgssc registered [12908.921931] Lustre: Echo OBD driver; http://www.lustre.org/ [12916.495830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12916.521514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12917.961273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12922.032976] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12927.462412] LustreError: 428115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12927.483532] LustreError: 428115:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12931.733797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12932.120083] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12936.943871] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12944.315943] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12949.779923] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12956.716137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12957.164122] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12962.302360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12963.760841] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [12971.319294] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12977.636573] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12977.647298] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12977.661030] Lustre: Skipped 1 previous similar message [12977.676156] Lustre: Skipped 1 previous similar message [12978.481169] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.43@tcp (stopping) [12982.764036] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12982.778216] Lustre: Skipped 1 previous similar message [12987.880782] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12987.898788] Lustre: Skipped 2 previous similar messages [12989.407206] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12989.766755] Lustre: server umount lustre-MDT0001 complete [12992.994087] LustreError: 428114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12993.008359] LustreError: 428114:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12998.118512] LustreError: 428115:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12998.120641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12998.159433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12998.179662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12998.196334] Lustre: Skipped 1 previous similar message [13003.235183] LustreError: 428114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13003.825566] Lustre: server umount lustre-OST0000 complete [13008.795768] Lustre: server umount lustre-MDT0000 complete [13025.062470] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [13028.889524] Key type lgssc unregistered [13029.252198] LNet: 431801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13029.267112] LNetError: 431801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13029.283068] LNet: Removed LNI 192.168.204.143@tcp [13030.060765] Key type .llcrypt unregistered [13030.063535] Key type ._llcrypt unregistered [13055.691491] Key type ._llcrypt registered [13055.695368] Key type .llcrypt registered [13055.803909] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [13071.580965] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [13072.620613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13072.793206] alg: No test for adler32 (adler32-zlib) [13073.964086] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [13074.228327] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [13075.920125] Key type lgssc registered [13077.436141] Lustre: Echo OBD driver; http://www.lustre.org/ [13117.625791] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [13137.453212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13137.488250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13138.910824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13138.941873] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13139.125305] Lustre: lustre-MDT0000: new disk, initializing [13139.311708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13139.327239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13145.122648] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13162.295627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13162.526446] Lustre: lustre-OST0000: new disk, initializing [13162.531384] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13162.535639] Lustre: Skipped 1 previous similar message [13162.540677] Lustre: 436832:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13162.613201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13169.254273] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13169.262891] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13169.301335] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13169.942532] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13179.828442] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13195.795833] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13195.914266] Lustre: 435816:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13195.977716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13196.071732] Lustre: lustre-MDT0001: new disk, initializing [13196.295590] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13196.331785] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13196.341231] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13201.222163] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13211.600455] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13211.843368] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13217.293675] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13217.582576] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13222.734151] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.43@tcp (stopping) [13225.956363] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13225.958271] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13225.979443] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13226.002641] Lustre: Skipped 1 previous similar message [13227.830786] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.43@tcp (stopping) [13227.836726] Lustre: Skipped 1 previous similar message [13228.964948] Lustre: server umount lustre-MDT0001 complete [13236.194159] LustreError: 435827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.213405] LustreError: 435827:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [13241.316368] LustreError: 435826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13241.360818] LustreError: 435826:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13246.436988] LustreError: 438163:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13246.476867] LustreError: 438163:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13251.041044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13251.048068] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13251.056320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13251.554020] LustreError: 435827:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13253.803281] Lustre: server umount lustre-OST0000 complete [13256.677942] LustreError: 435826:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13265.199711] Lustre: server umount lustre-MDT0000 complete [13281.059055] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [13284.126848] Key type lgssc unregistered [13284.404517] LNet: 440580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13284.412164] LNetError: 440580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13284.434945] LNet: Removed LNI 192.168.204.143@tcp [13285.318927] Key type .llcrypt unregistered [13285.323444] Key type ._llcrypt unregistered [13337.495256] Key type ._llcrypt registered [13337.498785] Key type .llcrypt registered [13337.665872] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [13337.973384] Key type .llcrypt unregistered [13337.975192] Key type ._llcrypt unregistered [13359.262853] Key type ._llcrypt registered [13359.265596] Key type .llcrypt registered [13359.423105] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [13360.975385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13361.149540] alg: No test for adler32 (adler32-zlib) [13362.540927] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [13363.046131] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [13364.888183] Key type lgssc registered [13367.127808] Lustre: Echo OBD driver; http://www.lustre.org/ [13400.064992] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13400.130488] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13401.442734] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13401.475677] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13401.587528] Lustre: lustre-MDT0000: new disk, initializing [13401.705750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13401.731899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13406.555736] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13418.777773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13419.046635] Lustre: 443952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13419.127602] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13419.131276] Lustre: Skipped 1 previous similar message [13419.238404] Lustre: lustre-MDT0001: new disk, initializing [13419.407278] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13419.448579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13419.461427] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13425.058771] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13438.129597] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13447.501602] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13455.340475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13455.621951] Lustre: lustre-OST0000: new disk, initializing [13455.625148] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13455.632931] Lustre: 445925:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13455.733985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13462.172569] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13464.648677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13464.661430] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13464.781162] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13472.751259] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13480.432540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13480.445141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13480.475371] Lustre: Skipped 1 previous similar message [13480.519243] Lustre: Skipped 2 previous similar messages [13485.551321] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13485.563917] Lustre: Skipped 2 previous similar messages [13486.338891] Lustre: server umount lustre-MDT0000 complete [13490.992189] LustreError: 443946:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789615397 with bad export cookie 12981077206070557924 [13491.003293] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13491.379225] Lustre: server umount lustre-MDT0001 complete [13506.307451] Lustre: server umount lustre-OST0000 complete [13519.169377] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:23:43 (1789615423) [13527.215156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13527.710663] LustreError: 447957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13527.739443] LustreError: 447957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [13527.800581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13532.257751] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13533.154663] LustreError: 447958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13537.273446] LustreError: 447957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13541.413387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13541.795198] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13546.350098] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13554.458503] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13559.792430] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13580.973852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13581.326822] Lustre: Found index 0 for lustre-OST0000, updating log [13581.480721] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13587.914042] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [13597.080583] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13598.906853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13609.443703] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13609.444841] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13609.455607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13609.463063] Lustre: Skipped 2 previous similar messages [13613.757851] Lustre: server umount lustre-OST0000 complete [13631.976151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13631.983714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13632.012705] Lustre: Skipped 1 previous similar message [13632.026260] Lustre: Skipped 4 previous similar messages [13637.090827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13637.111791] Lustre: Skipped 1 previous similar message [13639.647161] LNet: 1 local NIs in recovery (showing 1): 192.168.204.143@tcp [13645.279150] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13645.546960] Lustre: server umount lustre-MDT0000 complete [13649.380269] LustreError: 447954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13649.412286] LustreError: 447954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13650.418310] LustreError: 447937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789615557 with bad export cookie 12981077206070559317 [13650.437325] LustreError: 447937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [13650.441431] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13651.164804] Lustre: server umount lustre-MDT0001 complete [13663.010352] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:26:07 (1789615567) [14295.830095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14296.570174] LustreError: 454373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14296.719201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14302.177587] LustreError: 454374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14302.879630] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [14306.282396] LustreError: 454373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14312.433591] LustreError: 454373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14312.466563] LustreError: 454373:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14312.822284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14313.237216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14319.700596] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [14328.647889] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14335.913303] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14355.487343] LNet: 1 local NIs in recovery (showing 1): 192.168.204.143@tcp [14358.398077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14358.798255] Lustre: Found index 0 for lustre-OST0000, updating log [14359.223930] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14369.339816] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [14379.201721] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14383.210784] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14383.226532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14383.236716] Lustre: Skipped 6 previous similar messages [14388.494279] Lustre: server umount lustre-OST0000 complete [14394.209084] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14394.224437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14394.226965] Lustre: Skipped 1 previous similar message [14394.250793] Lustre: Skipped 1 previous similar message [14396.323979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14396.328250] Lustre: Skipped 3 previous similar messages [14399.527350] Lustre: server umount lustre-MDT0000 complete [14400.418752] LustreError: 454370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14400.428560] LustreError: 454370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14403.239843] LustreError: 454355:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789616310 with bad export cookie 12981077206070560402 [14403.254519] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14403.691848] Lustre: server umount lustre-MDT0001 complete [14415.905730] LNet: 1 local NIs in recovery (showing 1): 192.168.204.143@tcp [14441.242494] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [14452.365188] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [14500.294976] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:40:04 (1789616404) [14523.307927] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [14527.127624] Key type lgssc unregistered [14527.698140] LNet: 460994:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14527.714894] LNetError: 460994:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14619.883804] LNet: Removed LNI 192.168.204.143@tcp [14620.896886] Key type .llcrypt unregistered [14620.899110] Key type ._llcrypt unregistered [14639.093343] Key type ._llcrypt registered [14639.095381] Key type .llcrypt registered [14639.242342] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [14640.482923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14640.629397] alg: No test for adler32 (adler32-zlib) [14641.685688] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [14641.922069] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [14643.639440] Key type lgssc registered [14644.861399] Lustre: Echo OBD driver; http://www.lustre.org/ [14649.492986] LNet: Added LNI 192.168.204.143@tcp100 [8/256/0/180] [14650.341202] LNet: Added LNI 192.168.204.143@tcp101 [8/256/0/180] [14651.167115] LNet: Added LNI 192.168.204.143@tcp102 [8/256/0/180] [14652.048760] LNet: Added LNI 192.168.204.143@tcp103 [8/256/0/180] [14652.961174] LNet: Added LNI 192.168.204.143@tcp104 [8/256/0/180] [14653.896698] LNet: Added LNI 192.168.204.143@tcp105 [8/256/0/180] [14654.861669] LNet: Added LNI 192.168.204.143@tcp106 [8/256/0/180] [14655.629418] LNet: Added LNI 192.168.204.143@tcp107 [8/256/0/180] [14656.453140] LNet: Added LNI 192.168.204.143@tcp108 [8/256/0/180] [14657.319539] LNet: Added LNI 192.168.204.143@tcp109 [8/256/0/180] [14658.153537] LNet: Added LNI 192.168.204.143@tcp110 [8/256/0/180] [14658.991587] LNet: Added LNI 192.168.204.143@tcp111 [8/256/0/180] [14659.877404] LNet: Added LNI 192.168.204.143@tcp112 [8/256/0/180] [14661.116120] LNet: Added LNI 192.168.204.143@tcp113 [8/256/0/180] [14662.111244] LNet: Added LNI 192.168.204.143@tcp114 [8/256/0/180] [14663.046985] LNet: Added LNI 192.168.204.143@tcp115 [8/256/0/180] [14663.914655] LNet: Added LNI 192.168.204.143@tcp116 [8/256/0/180] [14664.778414] LNet: Added LNI 192.168.204.143@tcp117 [8/256/0/180] [14665.656633] LNet: Added LNI 192.168.204.143@tcp118 [8/256/0/180] [14666.577988] LNet: Added LNI 192.168.204.143@tcp119 [8/256/0/180] [14667.422655] LNet: Added LNI 192.168.204.143@tcp120 [8/256/0/180] [14668.379571] LNet: Added LNI 192.168.204.143@tcp121 [8/256/0/180] [14669.228360] LNet: Added LNI 192.168.204.143@tcp122 [8/256/0/180] [14670.059606] LNet: Added LNI 192.168.204.143@tcp123 [8/256/0/180] [14670.951680] LNet: Added LNI 192.168.204.143@tcp124 [8/256/0/180] [14671.945402] LNet: Added LNI 192.168.204.143@tcp125 [8/256/0/180] [14672.983466] LNet: Added LNI 192.168.204.143@tcp126 [8/256/0/180] [14674.074294] LNet: Added LNI 192.168.204.143@tcp127 [8/256/0/180] [14675.191857] LNet: Added LNI 192.168.204.143@tcp128 [8/256/0/180] [14676.250110] LNet: Added LNI 192.168.204.143@tcp129 [8/256/0/180] [14677.599976] LNet: Added LNI 192.168.204.143@tcp130 [8/256/0/180] [14678.584992] LNet: Added LNI 192.168.204.143@tcp131 [8/256/0/180] [14679.852526] LNet: Added LNI 192.168.204.143@tcp132 [8/256/0/180] [14680.903968] LNet: Added LNI 192.168.204.143@tcp133 [8/256/0/180] [14681.994687] LNet: Added LNI 192.168.204.143@tcp134 [8/256/0/180] [14683.031594] LNet: Added LNI 192.168.204.143@tcp135 [8/256/0/180] [14692.919579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14692.969329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14694.222747] Lustre: 464021:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.143@tcp: can't add NID 192.168.204.143@tcp130: rc = -75 [14694.372977] LNetError: 464041:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.143 [14694.414406] LNetError: Protocol error connecting to 192.168.204.143@tcp120 at host 192.168.204.143:988: is it running a compatible version of Lustre? [14694.444369] LNetError: Skipped 4 previous similar messages [14694.651686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14694.718544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14694.891456] Lustre: lustre-MDT0000: new disk, initializing [14695.079389] Lustre: 464078:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.143@tcp: can't add NID 192.168.204.143@tcp130: rc = -75 [14695.093903] Lustre: 464078:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14695.128644] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14695.162631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14695.200538] LNet: 7 local NIs in recovery (showing 7): 192.168.204.143@tcp123, 192.168.204.143@tcp124, 192.168.204.143@tcp125, 192.168.204.143@tcp126, 192.168.204.143@tcp127, 192.168.204.143@tcp120, 192.168.204.143@tcp129 [14695.205828] LNetError: 464044:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.143 [14695.263160] LNetError: 464044:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14695.290719] LNetError: Protocol error connecting to 192.168.204.143@tcp111 at host 192.168.204.143:988: is it running a compatible version of Lustre? [14695.312188] LNetError: Skipped 5 previous similar messages [14701.751355] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [14715.270670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14715.527835] Lustre: 464039:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14715.645873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14715.653587] Lustre: Skipped 1 previous similar message [14715.766311] Lustre: lustre-MDT0001: new disk, initializing [14715.812663] Lustre: 464841:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.143@tcp: can't add NID 192.168.204.143@tcp130: rc = -75 [14715.826464] Lustre: 464841:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14716.112791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14716.233201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14716.243261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14722.433692] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [14724.658263] Lustre: 465185:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.143@tcp: can't add NID 192.168.204.143@tcp130: rc = -75 [14724.673067] Lustre: 465185:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14724.962877] LNetError: 464047:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.143 [14724.993259] LNetError: Protocol error connecting to 192.168.204.143@tcp118 at host 192.168.204.143:988: is it running a compatible version of Lustre? [14727.013097] LNetError: 461475:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.143 [14727.021764] LNetError: Protocol error connecting to 192.168.204.143@tcp121 at host 192.168.204.143:988: is it running a compatible version of Lustre? [14731.108656] LNetError: 464052:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.143 [14731.123263] LNetError: 464052:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14731.138322] LNetError: Protocol error connecting to 192.168.204.143@tcp122 at host 192.168.204.143:988: is it running a compatible version of Lustre? [14731.150563] LNetError: Skipped 1 previous similar message [14733.916716] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14739.298621] LNetError: 461476:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.143 [14739.315154] LNetError: 461476:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [14739.322376] LNetError: Protocol error connecting to 192.168.204.143@tcp121 at host 192.168.204.143:988: is it running a compatible version of Lustre? [14739.336750] LNetError: Skipped 3 previous similar messages [14740.473113] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14748.622246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14748.936874] Lustre: lustre-OST0000: new disk, initializing [14748.940631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14748.947716] Lustre: 466015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14749.038567] Lustre: 466020:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.143@tcp: can't add NID 192.168.204.143@tcp130: rc = -75 [14749.047299] Lustre: 466020:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14749.128742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14755.692601] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [14757.728792] LNetError: 461474:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.143 [14757.739989] LNetError: 461474:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14757.748808] LNetError: Protocol error connecting to 192.168.204.143@tcp122 at host 192.168.204.143:988: is it running a compatible version of Lustre? [14757.755848] LNetError: Skipped 7 previous similar messages [14757.922538] Lustre: 466472:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.143@tcp: can't add NID 192.168.204.143@tcp130: rc = -75 [14757.932896] Lustre: 466472:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14757.998151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14758.010725] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14758.102345] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14764.425377] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14773.733111] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14773.736713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14773.740380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14773.748590] Lustre: Skipped 1 previous similar message [14777.303735] Lustre: server umount lustre-OST0000 complete [14782.436495] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14782.447502] LustreError: Skipped 1 previous similar message [14782.453705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14782.473032] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14782.480269] Lustre: Skipped 1 previous similar message [14783.969222] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14783.969420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14784.017264] Lustre: Skipped 1 previous similar message [14789.090467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14789.101162] Lustre: Skipped 1 previous similar message [14790.497091] LNetError: 464052:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.143 [14790.523940] LNetError: 464052:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [14790.538610] LNetError: Protocol error connecting to 192.168.204.143@tcp121 at host 192.168.204.143:988: is it running a compatible version of Lustre? [14790.552791] LNetError: Skipped 9 previous similar messages [14794.209638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14794.219457] Lustre: Skipped 1 previous similar message [14795.231181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14795.498994] Lustre: server umount lustre-MDT0000 complete [14799.527980] LustreError: 464030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789616706 with bad export cookie 8139822873767974170 [14799.544107] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14799.547214] LustreError: 464030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14799.998540] Lustre: server umount lustre-MDT0001 complete [14812.443385] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [14816.265217] Key type lgssc unregistered [14816.953367] LNet: 467929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14816.971419] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14816.982691] LNet: Removed LNI 192.168.204.143@tcp [14818.015215] LNet: Removed LNI 192.168.204.143@tcp100 [14818.031111] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14818.039212] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14819.103317] LNet: Removed LNI 192.168.204.143@tcp101 [14819.108040] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14819.117435] LNet: Removed LNI 192.168.204.143@tcp102 [14820.127266] LNet: Removed LNI 192.168.204.143@tcp103 [14821.151402] LNet: Removed LNI 192.168.204.143@tcp104 [14821.175628] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14821.194048] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14821.225974] LNet: Removed LNI 192.168.204.143@tcp105 [14822.303359] LNet: Removed LNI 192.168.204.143@tcp106 [14823.391424] LNet: Removed LNI 192.168.204.143@tcp107 [14824.415200] LNet: Removed LNI 192.168.204.143@tcp108 [14825.439208] LNet: Removed LNI 192.168.204.143@tcp109 [14825.445594] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14825.466836] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14825.491734] LNet: Removed LNI 192.168.204.143@tcp110 [14826.527180] LNet: Removed LNI 192.168.204.143@tcp111 [14827.551274] LNet: Removed LNI 192.168.204.143@tcp112 [14828.575982] LNet: Removed LNI 192.168.204.143@tcp113 [14829.599473] LNet: Removed LNI 192.168.204.143@tcp114 [14830.625288] LNet: Removed LNI 192.168.204.143@tcp115 [14831.648238] LNet: Removed LNI 192.168.204.143@tcp116 [14832.671196] LNet: Removed LNI 192.168.204.143@tcp117 [14833.695834] LNet: Removed LNI 192.168.204.143@tcp118 [14833.705237] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14833.717899] LNetError: 467929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14833.736951] LNet: Removed LNI 192.168.204.143@tcp119 [14834.783336] LNet: Removed LNI 192.168.204.143@tcp120 [14835.807374] LNet: Removed LNI 192.168.204.143@tcp121 [14836.831181] LNet: Removed LNI 192.168.204.143@tcp122 [14837.855289] LNet: Removed LNI 192.168.204.143@tcp123 [14838.879182] LNet: Removed LNI 192.168.204.143@tcp124 [14839.907244] LNet: Removed LNI 192.168.204.143@tcp125 [14840.928904] LNet: Removed LNI 192.168.204.143@tcp126 [14841.951160] LNet: Removed LNI 192.168.204.143@tcp127 [14842.975446] LNet: Removed LNI 192.168.204.143@tcp128 [14843.999357] LNet: Removed LNI 192.168.204.143@tcp129 [14844.011503] LNet: Removed LNI 192.168.204.143@tcp130 [14844.019596] LNet: Removed LNI 192.168.204.143@tcp131 [14844.029256] LNet: Removed LNI 192.168.204.143@tcp132 [14844.039509] LNet: Removed LNI 192.168.204.143@tcp133 [14844.044713] LNet: Removed LNI 192.168.204.143@tcp134 [14844.065152] LNet: Removed LNI 192.168.204.143@tcp135 [14845.054333] Key type .llcrypt unregistered [14845.058325] Key type ._llcrypt unregistered [14870.254956] Key type ._llcrypt registered [14870.259413] Key type .llcrypt registered [14870.437182] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [14885.977291] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [14887.529426] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14887.587795] alg: No test for adler32 (adler32-zlib) [14888.689559] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [14889.001353] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [14890.751539] Key type lgssc registered [14892.192966] Lustre: Echo OBD driver; http://www.lustre.org/ [14927.995195] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:47:12 (1789616832) [14947.495326] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [14950.898335] Key type lgssc unregistered [14951.190859] LNet: 472014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14951.207823] LNetError: 472014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14951.228426] LNet: Removed LNI 192.168.204.143@tcp [14952.097151] Key type .llcrypt unregistered [14952.101734] Key type ._llcrypt unregistered [14968.723394] Key type ._llcrypt registered [14968.726167] Key type .llcrypt registered [14968.819730] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [14969.660384] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14969.747559] alg: No test for adler32 (adler32-zlib) [14970.864669] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [14971.180398] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [14972.879285] Key type lgssc registered [14974.036574] Lustre: Echo OBD driver; http://www.lustre.org/ [14984.261801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14984.309972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14985.723505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14985.791715] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14985.943420] Lustre: lustre-MDT0000: new disk, initializing [14986.062871] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14986.092540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14991.064061] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15007.121702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15007.388867] Lustre: 473220:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15007.478075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15007.485196] Lustre: Skipped 1 previous similar message [15007.667950] Lustre: lustre-MDT0001: new disk, initializing [15007.882712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15007.994135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15008.032064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15013.572650] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15024.437658] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15030.592484] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15037.513782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15037.833716] Lustre: lustre-OST0000: new disk, initializing [15037.837860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15037.847415] Lustre: 475192:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15037.954569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15043.831708] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15044.210974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15044.220865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15044.376420] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15052.893742] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15056.595582] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing lnet_if_list [15062.373893] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15062.412305] LNet: Added LNI 192.168.204.143@tcp10 [8/256/0/180] [15069.729077] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15069.760428] LNet: Added LNI 192.168.204.143@tcp11 [8/256/0/180] [15074.330704] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15074.370332] LNet: Added LNI 192.168.204.143@tcp12 [8/256/0/180] [15079.480829] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15079.526253] LNet: Added LNI 192.168.204.143@tcp13 [8/256/0/180] [15084.448567] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15084.502179] LNet: Added LNI 192.168.204.143@tcp14 [8/256/0/180] [15089.636435] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15089.697626] LNet: Added LNI 192.168.204.143@tcp15 [8/256/0/180] [15094.735922] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15094.780341] LNet: Added LNI 192.168.204.143@tcp16 [8/256/0/180] [15100.807962] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15100.852624] LNet: Added LNI 192.168.204.143@tcp17 [8/256/0/180] [15106.987397] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15107.017411] LNet: Added LNI 192.168.204.143@tcp18 [8/256/0/180] [15113.034351] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15113.057707] LNet: Added LNI 192.168.204.143@tcp19 [8/256/0/180] [15118.746282] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15118.768285] LNet: Added LNI 192.168.204.143@tcp20 [8/256/0/180] [15124.402406] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15124.435385] LNet: Added LNI 192.168.204.143@tcp21 [8/256/0/180] [15129.726282] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15129.758073] LNet: Added LNI 192.168.204.143@tcp22 [8/256/0/180] [15135.407207] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15135.452862] LNet: Added LNI 192.168.204.143@tcp23 [8/256/0/180] [15140.970860] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15141.000946] LNet: Added LNI 192.168.204.143@tcp24 [8/256/0/180] [15146.507342] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15146.550925] LNet: Added LNI 192.168.204.143@tcp25 [8/256/0/180] [15152.689424] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15152.737621] LNet: Added LNI 192.168.204.143@tcp26 [8/256/0/180] [15158.292299] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15158.330906] LNet: Added LNI 192.168.204.143@tcp27 [8/256/0/180] [15164.539819] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15164.566622] LNet: Added LNI 192.168.204.143@tcp28 [8/256/0/180] [15173.913904] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15174.005649] LNet: Added LNI 192.168.204.143@tcp29 [8/256/0/180] [15181.895138] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15181.941695] LNet: Added LNI 192.168.204.143@tcp30 [8/256/0/180] [15190.101383] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15190.138783] LNet: Added LNI 192.168.204.143@tcp31 [8/256/0/180] [15199.466633] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15199.486644] LNet: Added LNI 192.168.204.143@tcp32 [8/256/0/180] [15207.455421] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15207.511445] LNet: Added LNI 192.168.204.143@tcp33 [8/256/0/180] [15214.398260] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15214.451374] LNet: Added LNI 192.168.204.143@tcp34 [8/256/0/180] [15221.337674] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15221.379102] LNet: Added LNI 192.168.204.143@tcp35 [8/256/0/180] [15228.597360] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15228.631233] LNet: Added LNI 192.168.204.143@tcp36 [8/256/0/180] [15235.107424] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15235.126189] LNet: Added LNI 192.168.204.143@tcp37 [8/256/0/180] [15243.268176] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15243.313120] LNet: Added LNI 192.168.204.143@tcp38 [8/256/0/180] [15251.713960] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15251.762239] LNet: Added LNI 192.168.204.143@tcp39 [8/256/0/180] [15260.822484] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15260.860974] LNet: Added LNI 192.168.204.143@tcp40 [8/256/0/180] [15269.727847] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15269.757415] LNet: Added LNI 192.168.204.143@tcp41 [8/256/0/180] [15277.671514] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15277.708887] LNet: Added LNI 192.168.204.143@tcp42 [8/256/0/180] [15284.491760] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15284.530680] LNet: Added LNI 192.168.204.143@tcp43 [8/256/0/180] [15291.278519] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15291.314215] LNet: Added LNI 192.168.204.143@tcp44 [8/256/0/180] [15299.977846] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15300.010674] LNet: Added LNI 192.168.204.143@tcp45 [8/256/0/180] [15307.706883] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15307.732638] LNet: Added LNI 192.168.204.143@tcp46 [8/256/0/180] [15315.504431] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15315.527061] LNet: Added LNI 192.168.204.143@tcp47 [8/256/0/180] [15325.099933] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15325.123044] LNet: Added LNI 192.168.204.143@tcp48 [8/256/0/180] [15332.884777] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15332.910877] LNet: Added LNI 192.168.204.143@tcp49 [8/256/0/180] [15342.016369] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15342.047358] LNet: Added LNI 192.168.204.143@tcp50 [8/256/0/180] [15362.019310] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15362.027683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15362.029379] Lustre: Skipped 1 previous similar message [15362.037533] Lustre: Skipped 2 previous similar messages [15367.136424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15367.160404] Lustre: Skipped 2 previous similar messages [15367.842101] Lustre: server umount lustre-MDT0000 complete [15372.004951] LustreError: 473213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617278 with bad export cookie 5279168215683571747 [15372.007825] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15372.019971] LustreError: 473213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15372.338386] Lustre: server umount lustre-MDT0001 complete [15388.639174] Lustre: 472495:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789617279/real 1789617279] req@0000000081ae4ed7 x1876549306595072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789617295 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15388.671882] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15388.694439] Lustre: Skipped 1 previous similar message [15390.209150] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [15393.759855] Lustre: 472496:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789617284/real 1789617284] req@00000000f3da9273 x1876549306595200/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789617300 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15397.856924] Lustre: 472493:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789617288/real 1789617288] req@000000007775a105 x1876549306595456/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789617304 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15398.598685] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15402.975299] Lustre: 472493:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789617293/real 1789617293] req@00000000e5cfdf59 x1876549306595584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789617309 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15416.353264] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x49435f859d5a2a79 [15416.362827] Lustre: MGC192.168.204.143@tcp: Connection restored to 0@lo (at 0@lo) [15416.659674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15417.700221] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15421.170907] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15423.986153] LustreError: 475186:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15424.012320] LustreError: 475186:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [15429.091333] LustreError: 475187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15430.282305] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15430.737763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15433.776270] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15435.918127] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15445.374435] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15451.695537] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15473.121658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15473.128881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15473.150151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15477.877978] Lustre: server umount lustre-OST0000 complete [15481.825737] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15481.841811] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15481.858918] Lustre: Skipped 1 previous similar message [15481.872206] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15481.878016] Lustre: Skipped 2 previous similar messages [15487.846927] Lustre: server umount lustre-MDT0000 complete [15490.020120] LustreError: 499439:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15490.036425] LustreError: 499439:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15491.223842] LustreError: 473215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617398 with bad export cookie 5279168215683574393 [15491.231423] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15491.236213] LustreError: 473215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15491.623350] Lustre: server umount lustre-MDT0001 complete [15501.588430] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [15504.470369] Key type lgssc unregistered [15504.866602] LNet: 502230:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15504.877718] LNetError: 502230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15505.889060] LNet: Removed LNI 192.168.204.143@tcp [15505.893581] LNetError: 502230:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15505.905592] LNet: Removed LNI 192.168.204.143@tcp10 [15505.910210] LNet: Removed LNI 192.168.204.143@tcp11 [15505.913452] LNet: Removed LNI 192.168.204.143@tcp12 [15505.916537] LNet: Removed LNI 192.168.204.143@tcp13 [15505.920812] LNet: Removed LNI 192.168.204.143@tcp14 [15505.926088] LNet: Removed LNI 192.168.204.143@tcp15 [15505.929014] LNet: Removed LNI 192.168.204.143@tcp16 [15505.932737] LNet: Removed LNI 192.168.204.143@tcp17 [15505.936411] LNet: Removed LNI 192.168.204.143@tcp18 [15505.942547] LNet: Removed LNI 192.168.204.143@tcp19 [15505.946643] LNet: Removed LNI 192.168.204.143@tcp20 [15505.950326] LNet: Removed LNI 192.168.204.143@tcp21 [15505.954549] LNet: Removed LNI 192.168.204.143@tcp22 [15505.963303] LNet: Removed LNI 192.168.204.143@tcp23 [15505.967850] LNet: Removed LNI 192.168.204.143@tcp24 [15505.972593] LNet: Removed LNI 192.168.204.143@tcp25 [15505.976481] LNet: Removed LNI 192.168.204.143@tcp26 [15505.980356] LNet: Removed LNI 192.168.204.143@tcp27 [15505.985358] LNet: Removed LNI 192.168.204.143@tcp28 [15505.987600] LNet: Removed LNI 192.168.204.143@tcp29 [15505.994846] LNet: Removed LNI 192.168.204.143@tcp30 [15505.997532] LNet: Removed LNI 192.168.204.143@tcp31 [15506.004157] LNet: Removed LNI 192.168.204.143@tcp32 [15506.012523] LNet: Removed LNI 192.168.204.143@tcp33 [15506.024982] LNet: Removed LNI 192.168.204.143@tcp34 [15506.040164] LNet: Removed LNI 192.168.204.143@tcp35 [15506.048879] LNet: Removed LNI 192.168.204.143@tcp36 [15506.051146] LNet: Removed LNI 192.168.204.143@tcp37 [15506.061410] LNet: Removed LNI 192.168.204.143@tcp38 [15506.068081] LNet: Removed LNI 192.168.204.143@tcp39 [15506.072796] LNet: Removed LNI 192.168.204.143@tcp40 [15506.074921] LNet: Removed LNI 192.168.204.143@tcp41 [15506.084942] LNet: Removed LNI 192.168.204.143@tcp42 [15506.090365] LNet: Removed LNI 192.168.204.143@tcp43 [15506.103266] LNet: Removed LNI 192.168.204.143@tcp44 [15506.114619] LNet: Removed LNI 192.168.204.143@tcp45 [15506.123488] LNet: Removed LNI 192.168.204.143@tcp46 [15506.135659] LNet: Removed LNI 192.168.204.143@tcp47 [15506.144212] LNet: Removed LNI 192.168.204.143@tcp48 [15506.150588] LNet: Removed LNI 192.168.204.143@tcp49 [15506.165222] LNet: Removed LNI 192.168.204.143@tcp50 [15506.908591] Key type .llcrypt unregistered [15506.914703] Key type ._llcrypt unregistered [15527.300886] Key type ._llcrypt registered [15527.303761] Key type .llcrypt registered [15527.442990] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [15542.757441] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [15543.938797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15544.039722] alg: No test for adler32 (adler32-zlib) [15545.253227] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [15545.688112] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [15547.407546] Key type lgssc registered [15548.554076] Lustre: Echo OBD driver; http://www.lustre.org/ [15587.223307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15587.274609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15588.497460] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15588.549463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15588.676657] Lustre: lustre-MDT0000: new disk, initializing [15588.801530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15588.841787] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15593.480649] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15604.009165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15604.136937] Lustre: 505423:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15604.232153] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15604.246127] Lustre: Skipped 1 previous similar message [15604.323441] Lustre: lustre-MDT0001: new disk, initializing [15604.410729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15604.453461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15604.468296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15608.761627] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15617.819448] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15624.266366] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15633.413555] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15633.787693] Lustre: *** cfs_fail_loc=90f, val=0*** [15633.914522] Lustre: lustre-OST0000: new disk, initializing [15633.917588] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15633.924867] Lustre: 507442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15634.034268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15640.695729] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15640.712592] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15640.945719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15642.289582] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15652.590179] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15661.540044] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15661.543195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15661.556687] LustreError: Skipped 1 previous similar message [15661.559271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15661.568646] Lustre: Skipped 1 previous similar message [15661.585756] Lustre: Skipped 1 previous similar message [15666.748764] Lustre: server umount lustre-OST0000 complete [15671.265125] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15671.272268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15671.291735] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15676.897871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15676.905743] Lustre: Skipped 1 previous similar message [15677.354487] Lustre: server umount lustre-MDT0000 complete [15682.024105] LustreError: 505415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617588 with bad export cookie 10190049148779162556 [15682.027515] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15682.046572] LustreError: 505415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15682.623980] Lustre: server umount lustre-MDT0001 complete [15696.111934] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [15699.005949] Key type lgssc unregistered [15699.311716] LNet: 509348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15699.330163] LNetError: 509348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15699.355145] LNet: Removed LNI 192.168.204.143@tcp [15700.258552] Key type .llcrypt unregistered [15700.261101] Key type ._llcrypt unregistered [15719.873672] Key type ._llcrypt registered [15719.875959] Key type .llcrypt registered [15719.989821] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [15721.113882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15721.280686] alg: No test for adler32 (adler32-zlib) [15722.287302] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [15722.502408] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [15724.167269] Key type lgssc registered [15725.263554] Lustre: Echo OBD driver; http://www.lustre.org/ [15733.411374] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15733.433975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15734.973058] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15739.656258] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15744.490742] LustreError: 510616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15744.504527] LustreError: 510616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15749.068802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15749.704040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15755.347425] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15764.989314] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15770.937208] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15778.700288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15779.083510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15786.552200] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15796.235941] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15809.886688] loop0: detected capacity change from 0 to 204800000 [15810.276646] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15810.329528] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15810.377957] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15817.911689] loop0: detected capacity change from 0 to 204800000 [15818.184413] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15818.213135] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15818.252782] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15823.953555] loop0: detected capacity change from 0 to 204800000 [15829.406467] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15829.530492] Lustre: fs15246-OST0000: new disk, initializing [15829.534192] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15829.538837] Lustre: 514227:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15829.579381] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15836.624836] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15845.757313] loop1: detected capacity change from 0 to 204800000 [15852.405918] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15852.553041] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15852.628660] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15852.759334] Lustre: fs15246-MDT0000: new disk, initializing [15852.971543] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15852.993975] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15853.105245] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15854.183459] Lustre: 515348:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000799779e9 x1876550094995712/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1789617776 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15857.440799] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [15863.287936] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15863.306425] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15863.343815] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15877.607656] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15877.625058] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15882.741844] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15883.275291] Lustre: server umount fs15246-MDT0000 complete [15887.860359] LustreError: 510613:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15892.962149] LustreError: 513720:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15893.701489] Lustre: server umount fs15246-OST0000 complete [15907.296561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15907.308583] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15907.315972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15908.331881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15908.342934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15912.231285] Lustre: server umount lustre-OST0000 complete [15918.562216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15918.572416] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15918.591932] Lustre: Skipped 1 previous similar message [15918.614232] Lustre: Skipped 3 previous similar messages [15923.682187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15923.686469] Lustre: Skipped 1 previous similar message [15931.359342] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15931.772583] Lustre: server umount lustre-MDT0000 complete [15933.924989] LustreError: 510616:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15933.939829] LustreError: 510616:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15936.197857] LustreError: 510597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789617842 with bad export cookie 7150268540212966680 [15936.206384] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15936.208795] LustreError: 510597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15936.579635] Lustre: server umount lustre-MDT0001 complete [15947.304768] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [15949.994777] Key type lgssc unregistered [15950.310832] LNet: 518446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15950.316741] LNetError: 518446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15950.332629] LNet: Removed LNI 192.168.204.143@tcp [15951.286800] Key type .llcrypt unregistered [15951.291204] Key type ._llcrypt unregistered [15969.949775] Key type ._llcrypt registered [15969.954305] Key type .llcrypt registered [15970.140865] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [15971.460711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15971.577948] alg: No test for adler32 (adler32-zlib) [15972.737956] Lustre: Lustre: Build Version: 2.17.58_39_ga3f6967 [15972.969446] LNet: Added LNI 192.168.204.143@tcp [8/256/0/180] [15974.631257] Key type lgssc registered [15975.801986] Lustre: Echo OBD driver; http://www.lustre.org/ [16011.348871] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [16020.134694] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [16058.124710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16058.146679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16059.521405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16059.541307] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16059.602496] Lustre: lustre-MDT0000: new disk, initializing [16059.689465] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16059.719462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16065.512245] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [16080.643567] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16080.911974] Lustre: 522912:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16080.963455] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16080.967420] Lustre: Skipped 1 previous similar message [16081.024619] Lustre: lustre-MDT0001: new disk, initializing [16081.110649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16081.147588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16081.158502] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16087.706365] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [16100.874580] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16108.715789] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16116.080819] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16116.544992] Lustre: lustre-OST0000: new disk, initializing [16116.547945] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16116.554596] Lustre: 524880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16116.638048] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16122.837547] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_default_debug -1 all [16123.993962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16124.003416] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16124.117432] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16131.973439] Lustre: DEBUG MARKER: oleg443-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16136.772685] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16137.034615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16142.086843] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16142.316442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16144.865909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16144.869606] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16144.883708] Lustre: Skipped 1 previous similar message [16149.993918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16150.011267] Lustre: Skipped 2 previous similar messages [16150.351923] Lustre: server umount lustre-OST0000 complete [16155.106997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16155.108404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16155.126982] Lustre: Skipped 1 previous similar message [16159.805414] Lustre: server umount lustre-MDT0000 complete [16162.953534] LustreError: 522902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789618069 with bad export cookie 3808169592684434224 [16162.960085] LustreError: MGC192.168.204.143@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16162.961662] LustreError: 522902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16163.266496] Lustre: server umount lustre-MDT0001 complete [16187.625532] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing set_hostid [16200.051737] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing load_modules_local [16238.512129] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15930 sec ========= 00:09:03 (1789618143) [16241.138633] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:09:05 (1789618145) === [16245.477285] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:09:09 (1789618149) === [16288.079545] Lustre: DEBUG MARKER: oleg443-server.virtnet: executing unload_modules_local [16291.293359] Key type lgssc unregistered [16291.552597] LNet: 531523:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16291.563195] LNetError: 531523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16291.585209] LNet: Removed LNI 192.168.204.143@tcp [16292.407618] Key type .llcrypt unregistered [16292.410121] Key type ._llcrypt unregistered