[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 929988900 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.996 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2592MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002333] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004011] kvm-guest: setup PV IPIs [ 0.007544] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399996) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010121] LSM: Security Framework initializing [ 0.011043] Yama: becoming mindful. [ 0.012031] SELinux: Initializing. [ 0.013052] *** VALIDATE selinux *** [ 0.021184] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025715] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026133] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027094] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029032] *** VALIDATE tmpfs *** [ 0.030391] *** VALIDATE proc *** [ 0.031201] *** VALIDATE cgroup *** [ 0.032007] *** VALIDATE cgroup2 *** [ 0.033234] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034141] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036025] Spectre V2 : User space: Vulnerable [ 0.037006] Speculative Store Bypass: Vulnerable [ 0.040453] debug: unmapping init [mem 0xffffffffba659000-0xffffffffba660fff] [ 0.043000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043633] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044020] ... version: 2 [ 0.045009] ... bit width: 48 [ 0.046008] ... generic registers: 4 [ 0.047009] ... value mask: 0000ffffffffffff [ 0.048010] ... max period: 00007fffffffffff [ 0.049011] ... fixed-purpose events: 3 [ 0.050009] ... event mask: 000000070000000f [ 0.051269] rcu: Hierarchical SRCU implementation. [ 0.053480] smp: Bringing up secondary CPUs ... [ 0.054629] x86: Booting SMP configuration: [ 0.055020] .... node #0, CPUs: #1 #2 #3 [ 0.059579] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062011] smpboot: Total of 4 processors activated (19199.96 BogoMIPS) [ 0.090014] node 0 deferred pages initialised in 26ms [ 0.093174] devtmpfs: initialized [ 0.094216] x86/mm: Memory block size: 128MB [ 0.096886] gcov: version magic: 0x41383552 [ 0.098252] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.099074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.100219] pinctrl core: initialized pinctrl subsystem [ 0.101161] [ 0.101731] ************************************************************* [ 0.102011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.103010] ** ** [ 0.104010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.105012] ** ** [ 0.106010] ** This means that this kernel is built to expose internal ** [ 0.107010] ** IOMMU data structures, which may compromise security on ** [ 0.108009] ** your system. ** [ 0.109012] ** ** [ 0.110010] ** If you see this message and you are not debugging the ** [ 0.111009] ** kernel, report this immediately to your vendor! ** [ 0.112009] ** ** [ 0.113010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.114010] ************************************************************* [ 0.115741] NET: Registered protocol family 16 [ 0.116500] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.117095] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.118053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.119735] cpuidle: using governor menu [ 0.121888] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.124489] PCI: Using configuration type 1 for base access [ 0.126137] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.135097] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.136018] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.137504] cryptd: max_cpu_qlen set to 1000 [ 0.138765] ACPI: Added _OSI(Module Device) [ 0.139000] ACPI: Added _OSI(Processor Device) [ 0.140011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.141054] ACPI: Added _OSI(Processor Aggregator Device) [ 0.146894] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.154425] ACPI: Interpreter enabled [ 0.155061] ACPI: PM: (supports S0 S3 S4 S5) [ 0.156019] ACPI: Using IOAPIC for interrupt routing [ 0.161068] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.165503] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.269000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.269000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.269000] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.269000] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.269000] acpiphp: Slot [2] registered [ 0.269000] acpiphp: Slot [5] registered [ 0.269000] acpiphp: Slot [6] registered [ 0.269000] acpiphp: Slot [7] registered [ 0.269000] acpiphp: Slot [8] registered [ 0.269000] acpiphp: Slot [9] registered [ 0.269000] acpiphp: Slot [10] registered [ 0.269000] acpiphp: Slot [3] registered [ 0.269000] acpiphp: Slot [4] registered [ 0.269000] acpiphp: Slot [11] registered [ 0.269000] acpiphp: Slot [12] registered [ 0.269000] acpiphp: Slot [13] registered [ 0.269000] acpiphp: Slot [14] registered [ 0.269000] acpiphp: Slot [15] registered [ 0.269000] acpiphp: Slot [16] registered [ 0.269000] acpiphp: Slot [17] registered [ 0.269000] acpiphp: Slot [18] registered [ 0.269000] acpiphp: Slot [19] registered [ 0.269000] acpiphp: Slot [20] registered [ 0.269000] acpiphp: Slot [21] registered [ 0.269000] acpiphp: Slot [22] registered [ 0.269000] acpiphp: Slot [23] registered [ 0.269000] acpiphp: Slot [24] registered [ 0.269000] acpiphp: Slot [25] registered [ 0.269000] acpiphp: Slot [26] registered [ 0.269000] acpiphp: Slot [27] registered [ 0.269000] acpiphp: Slot [28] registered [ 0.269000] acpiphp: Slot [29] registered [ 0.269000] acpiphp: Slot [30] registered [ 0.269000] acpiphp: Slot [31] registered [ 0.269000] PCI host bridge to bus 0000:00 [ 0.269000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.269000] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.269000] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.269000] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.269000] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.269000] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.269000] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.269000] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.269000] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.269000] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.269000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.269000] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.278016] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.279000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.285142] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.286945] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.290042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.293988] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.299013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.314000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.314000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.314000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.314000] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.314000] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.314000] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.314000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.314000] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.314000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.321000] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.321000] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.321000] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.321000] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.374023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.423000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.432017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.441016] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.460016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.473239] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.544022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.570017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.600023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.602238] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.608016] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.609082] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.613000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.613000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.613000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.613000] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.613000] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.616292] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.622028] iommu: Default domain type: Passthrough [ 0.780673] SCSI subsystem initialized [ 0.783272] ACPI: bus type USB registered [ 0.785163] usbcore: registered new interface driver usbfs [ 0.797146] usbcore: registered new interface driver hub [ 0.799068] usbcore: registered new device driver usb [ 0.802315] pps_core: LinuxPPS API ver. 1 registered [ 0.803032] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.807056] PTP clock support registered [ 0.809313] EDAC MC: Ver: 3.0.0 [ 0.811130] PCI: Using ACPI for IRQ routing [ 0.812748] NetLabel: Initializing [ 0.814008] NetLabel: domain hash size = 128 [ 0.815007] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.816255] NetLabel: unlabeled traffic allowed by default [ 0.818250] vgaarb: loaded [ 0.820289] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.821009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.831119] clocksource: Switched to clocksource kvm-clock [ 0.947542] VFS: Disk quotas dquot_6.6.0 [ 0.948895] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.951213] *** VALIDATE ramfs *** [ 0.952444] *** VALIDATE hugetlbfs *** [ 0.954405] pnp: PnP ACPI init [ 0.956816] pnp: PnP ACPI: found 6 devices [ 0.971635] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.974537] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.976401] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.978279] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.980305] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.982389] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.984716] NET: Registered protocol family 2 [ 0.986935] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.991476] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.994705] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.999525] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.002862] TCP: Hash tables configured (established 65536 bind 65536) [ 1.009836] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.012589] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.015248] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.018386] NET: Registered protocol family 1 [ 1.020777] RPC: Registered named UNIX socket transport module. [ 1.023237] RPC: Registered udp transport module. [ 1.024902] RPC: Registered tcp transport module. [ 1.026626] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.029024] NET: Registered protocol family 44 [ 1.030635] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.032835] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.034843] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.037295] PCI: CLS 0 bytes, default 64 [ 1.038766] Unpacking initramfs... [ 2.866415] debug: unmapping init [mem 0xffff8cddbcc54000-0xffff8cddbffbffff] [ 2.870881] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.873500] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.876997] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229833f6470, max_idle_ns: 440795327230 ns [ 3.767081] Initialise system trusted keyrings [ 3.919730] Key type blacklist registered [ 3.923553] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.942282] zbud: loaded [ 3.946101] *** VALIDATE nfs *** [ 3.947513] *** VALIDATE nfs4 *** [ 3.949720] pstore: using deflate compression [ 3.953671] Platform Keyring initialized [ 4.080255] NET: Registered protocol family 38 [ 4.082025] Key type asymmetric registered [ 4.083949] Asymmetric key parser 'x509' registered [ 4.086335] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.089789] io scheduler mq-deadline registered [ 4.091758] io scheduler kyber registered [ 4.094769] io scheduler bfq registered [ 4.096762] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.101413] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.104513] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.107850] ACPI: Power Button [PWRF] [ 4.113826] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.122663] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.135530] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.145602] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.161224] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.191965] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.222101] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.227742] Non-volatile memory driver v1.3 [ 4.229923] Linux agpgart interface v0.103 [ 4.267876] virtio_blk virtio1: [vda] 139504 512-byte logical blocks (71.4 MB/68.1 MiB) [ 4.271986] vda: detected capacity change from 0 to 71426048 [ 4.289947] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.293713] vdb: detected capacity change from 0 to 1073741824 [ 4.314485] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.318735] vdc: detected capacity change from 0 to 2621440000 [ 4.341472] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.345506] vdd: detected capacity change from 0 to 2621440000 [ 4.364890] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.369147] vde: detected capacity change from 0 to 4294967296 [ 4.405828] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.409776] vdf: detected capacity change from 0 to 4294967296 [ 4.420984] libphy: Fixed MDIO Bus: probed [ 4.439405] usbcore: registered new interface driver usbserial_generic [ 4.441526] usbserial: USB Serial support registered for generic [ 4.444049] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.454199] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.456418] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.459706] mousedev: PS/2 mouse device common for all mice [ 4.463970] rtc_cmos 00:05: RTC can wake from S4 [ 4.468820] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.471513] rtc_cmos 00:05: registered as rtc0 [ 4.478835] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.482502] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.486027] intel_pstate: CPU model not supported [ 4.498223] hid: raw HID events driver (C) Jiri Kosina [ 4.500535] usbcore: registered new interface driver usbhid [ 4.502785] usbhid: USB HID core driver [ 4.504571] drop_monitor: Initializing network drop monitor service [ 4.507564] Initializing XFRM netlink socket [ 4.509374] NET: Registered protocol family 10 [ 4.512980] Segment Routing with IPv6 [ 4.514746] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.519399] NET: Registered protocol family 17 [ 4.521943] mpls_gso: MPLS GSO support [ 4.528459] RAS: Correctable Errors collector initialized. [ 4.530558] AVX version of gcm_enc/dec engaged. [ 4.532527] AES CTR mode by8 optimization enabled [ 4.650480] sched_clock: Marking stable (4650404983, 0)->(6549579886, -1899174903) [ 4.654782] registered taskstats version 1 [ 4.657541] Loading compiled-in X.509 certificates [ 4.659617] zswap: loaded using pool lzo/zbud [ 4.709680] Key type big_key registered [ 4.731451] Key type encrypted registered [ 4.733670] ima: No TPM chip found, activating TPM-bypass! [ 4.736974] ima: Allocated hash algorithm: sha1 [ 4.739614] ima: No architecture policies found [ 4.742312] evm: Initialising EVM extended attributes: [ 4.744822] evm: security.selinux [ 4.746767] evm: security.ima [ 4.748490] evm: security.capability [ 4.749991] evm: HMAC attrs: 0x1 [ 4.753527] rtc_cmos 00:05: setting system clock to 2026-06-26 14:41:48 UTC (1782484908) [ 4.763554] debug: unmapping init [mem 0xffffffffbb603000-0xffffffffbb7fffff] [ 4.767953] debug: unmapping init [mem 0xffffffffba382000-0xffffffffba658fff] [ 4.776257] Write protecting the kernel read-only data: 28672k [ 4.781490] debug: unmapping init [mem 0xffffffffb8a03000-0xffffffffb8bfffff] [ 4.784917] debug: unmapping init [mem 0xffffffffb9314000-0xffffffffb93fffff] [ 4.833401] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.839718] systemd[1]: Detected virtualization kvm. [ 4.841463] systemd[1]: Detected architecture x86-64. [ 4.843392] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.884490] systemd[1]: No hostname configured. [ 4.899551] systemd[1]: Set hostname to . [ 4.903227] random: systemd: uninitialized urandom read (16 bytes read) [ 4.905584] systemd[1]: Initializing machine ID from random generator. [ 5.106607] random: ln: uninitialized urandom read (6 bytes read) [ 5.310578] random: systemd: uninitialized urandom read (16 bytes read) [ 5.319426] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.336368] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.346816] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.427291] device-mapper: uevent: version 1.0.3 [ 7.429758] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ 9.380978] random: fast init done [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 9.841478] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 10.183145] scsi host0: ata_piix [ 10.502823] scsi host1: ata_piix [ 10.509628] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 10.526052] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 11.222006] hrtimer: interrupt took 6994735 ns [ 16.265774] random: crng init done [ 16.267365] random: 7 urandom warning(s) missed due to ratelimiting [ 19.600953] dracut-initqueue[582]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK [ 22.960480] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 26.611530] printk: systemd: 26 output lines suppressed due to ratelimiting [ 27.284994] SELinux: Disabled at runtime. [ 27.388681] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 27.417084] systemd[1]: Detected virtualization kvm. [ 27.418951] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 28.924041] systemd[1]: initrd-switch-root.service: Succeeded. [ 28.929393] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 28.962546] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 28.974761] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 28.984172] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 29.018387] systemd[1]: Starting Journal Service... Starting Journal Service... [ 29.054755] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 29.478096] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Mounting Kernel Debug File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 31.002050] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 32.697465] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 32.771971] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 33.484261] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 34.321334] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit) [ ***] A start job is running for Configur…only root support (11s / no limit) [ **] A start job is running for Configur…only root support (11s / no limit) [ *] A start job is running for Configur…only root support (12s / no limit) [ **] A start job is running for Configur…only root support (12s / no limit)[ 41.691086] Key type dns_resolver registered [ ***] A start job is running for Configur…only root support (13s / no limit) [ *** ] A start job is running for Configur…only root support (13s / no limit)[ 42.719581] NFS: Registering the id_resolver key type [ 42.726210] Key type id_resolver registered [ 42.727873] Key type id_legacy registered [ *** ] A start job is running for Configur…only root support (14s / no limit) [*** ] A start job is running for Configur…only root support (14s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Login Service... Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg228-server login: [ 110.853718] libcfs: loading out-of-tree module taints kernel. [ 110.894504] Key type ._llcrypt registered [ 110.896745] Key type .llcrypt registered [ 111.003738] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 127.391766] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 128.885823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 128.900368] alg: No test for adler32 (adler32-zlib) [ 130.240953] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 131.099168] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 132.776525] Key type lgssc registered [ 134.630403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 147.973964] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 180.090697] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 190.471784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 190.501633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 191.746507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 191.781168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 191.872415] Lustre: lustre-MDT0000: new disk, initializing [ 191.941650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 191.964564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 194.997669] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 205.086065] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 205.166433] Lustre: 6511:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 205.188975] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 205.193278] Lustre: Skipped 1 previous similar message [ 205.254801] Lustre: lustre-MDT0001: new disk, initializing [ 205.304109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 205.325452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 205.338138] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 208.427315] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.430394] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 219.969834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.172356] Lustre: lustre-OST0000: new disk, initializing [ 220.175650] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 220.180486] Lustre: 8447:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.233421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 222.027144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 222.037970] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 222.064610] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 225.027591] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 237.482875] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 237.578926] Lustre: lustre-OST0001: new disk, initializing [ 237.580990] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 237.583614] Lustre: 9518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 237.629356] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 243.538477] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 244.731718] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 244.738076] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 244.805724] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 253.093860] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 257.909051] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 262.932548] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing check_logdir /tmp/testlogs/ [ 266.599493] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing yml_node [ 269.837733] Lustre: DEBUG MARKER: Client: 2.17.54.4 [ 271.955474] Lustre: DEBUG MARKER: MDS: 2.17.54.4 [ 273.958301] Lustre: DEBUG MARKER: OSS: 2.17.54.4 [ 275.246289] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Jun 26 10:46:18 EDT 2026 [ 288.903476] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 290.126580] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 306.146125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.149250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 306.159237] Lustre: Skipped 1 previous similar message [ 306.167360] Lustre: Skipped 3 previous similar messages [ 312.853539] Lustre: server umount lustre-MDT0000 complete [ 315.361093] LustreError: 7923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485219 with bad export cookie 11591702540919948345 [ 315.369186] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 315.372096] LustreError: 7923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 316.387263] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.390473] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 316.412746] LustreError: 6518:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 316.424058] Lustre: Skipped 2 previous similar messages [ 316.427619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 316.431572] Lustre: Skipped 1 previous similar message [ 320.417299] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.417560] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.434898] LustreError: 6517:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 320.456960] Lustre: Skipped 1 previous similar message [ 321.811862] Lustre: server umount lustre-MDT0001 complete [ 325.298601] Lustre: server umount lustre-OST0000 complete [ 328.136664] Lustre: server umount lustre-OST0001 complete [ 332.608818] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 339.455113] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 371.100029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.375451] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 371.447841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.574888] Lustre: lustre-MDT0000: new disk, initializing [ 371.663790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.686337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 375.520888] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 386.382641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 386.500259] Lustre: 15070:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 386.573130] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 386.577143] Lustre: Skipped 1 previous similar message [ 386.647560] Lustre: lustre-MDT0001: new disk, initializing [ 386.791070] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 386.875948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 386.904034] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 391.007319] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 399.530427] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 404.426200] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 410.335210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 410.544960] Lustre: lustre-OST0000: new disk, initializing [ 410.547704] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 410.552821] Lustre: 17036:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 410.630136] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.350400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 412.362859] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 412.482233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.749609] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 424.987174] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 428.779107] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 428.961754] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 433.164357] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 433.348480] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 438.245579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 438.257250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 438.268498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 438.274724] Lustre: Skipped 2 previous similar messages [ 438.293160] Lustre: Skipped 1 previous similar message [ 441.523587] Lustre: server umount lustre-OST0000 complete [ 448.483986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 448.489100] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 448.508470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 448.526748] LustreError: Skipped 1 previous similar message [ 458.723742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 458.734820] Lustre: Skipped 4 previous similar messages [ 459.232130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 459.391802] Lustre: server umount lustre-MDT0000 complete [ 462.285085] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485366 with bad export cookie 11591702540919949878 [ 462.287691] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 462.295647] LustreError: 15063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 462.539223] Lustre: server umount lustre-MDT0001 complete [ 465.635153] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:49:29 (1782485369) [ 467.616867] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 469.139679] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:49:32 (1782485372) [ 474.295290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 474.615192] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 474.674323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 478.069279] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 479.714309] LustreError: 19447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 484.843675] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 485.180766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.371564] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 495.519394] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 500.322980] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 505.677444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 511.263384] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 517.953745] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 529.899326] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 529.901627] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 529.909300] Lustre: Skipped 1 previous similar message [ 529.910821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 535.296534] Lustre: server umount lustre-OST0000 complete [ 540.135237] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 540.162577] Lustre: Skipped 2 previous similar messages [ 545.142269] Lustre: server umount lustre-MDT0000 complete [ 545.250246] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.258365] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 548.311894] LustreError: 19426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485452 with bad export cookie 11591702540919950760 [ 548.321413] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.323221] LustreError: 19426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 548.516700] Lustre: server umount lustre-MDT0001 complete [ 558.663368] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 561.002411] Key type lgssc unregistered [ 561.217693] LNet: 22978:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 561.226981] LNetError: 22978:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 561.239994] LNet: Removed LNI 192.168.202.128@tcp [ 561.976162] Key type .llcrypt unregistered [ 561.977770] Key type ._llcrypt unregistered [ 581.481212] Key type ._llcrypt registered [ 581.482739] Key type .llcrypt registered [ 581.610630] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 582.357112] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 582.385216] alg: No test for adler32 (adler32-zlib) [ 583.372678] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 583.620077] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 585.248147] Key type lgssc registered [ 586.084374] Lustre: Echo OBD driver; http://www.lustre.org/ [ 592.732572] loop: module loaded [ 592.803110] loop0: detected capacity change from 0 to 204800000 [ 592.905461] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 592.915143] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.133438] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 598.239938] loop0: detected capacity change from 0 to 204800000 [ 598.453600] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 598.465928] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 598.475257] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 602.676381] loop0: detected capacity change from 0 to 204800000 [ 607.453540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 607.481318] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 608.693879] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 608.734517] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 608.811609] Lustre: lustre-MDT0000: new disk, initializing [ 608.897405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 608.913504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 613.979706] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 621.984788] loop1: detected capacity change from 0 to 204800000 [ 626.876919] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 627.079525] Lustre: lustre-OST07c6: new disk, initializing [ 627.082373] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 627.087789] Lustre: Skipped 1 previous similar message [ 627.091843] Lustre: 26161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 627.135933] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 632.320488] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 633.410761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 633.413953] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 633.441444] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 637.146314] Lustre: 25147:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 637.152735] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 637.157558] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 637.162912] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 637.168244] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 637.174273] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 652.558930] Lustre: 25147:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 652.564168] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 652.572652] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 652.576204] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 652.583983] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 652.591052] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 657.076261] Lustre: 25147:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 657.093696] Lustre: 25147:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 657.100286] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 657.107184] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.116308] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 657.123746] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.135446] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 657.143571] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.152529] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 657.156923] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 657.161368] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 657.172891] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.331272] Lustre: 25147:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 660.346695] Lustre: 25147:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 660.351618] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 660.355626] Lustre: 25147:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.361560] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 660.371069] Lustre: 25147:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.376888] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 660.388357] Lustre: 25147:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.399629] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 660.411292] Lustre: 25147:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 660.416364] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 660.420314] Lustre: 25147:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.746134] Lustre: 25148:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 664.760517] Lustre: 25148:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 664.765130] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 664.770197] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.780636] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 664.790153] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.804284] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 664.817508] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.831092] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 664.846838] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 664.854547] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 664.861949] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 675.593453] Lustre: 25148:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 675.606640] Lustre: 25148:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 675.619482] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 675.625762] Lustre: 25148:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 675.630884] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 675.636642] Lustre: 25148:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 675.642821] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 675.648551] Lustre: 25148:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 675.653457] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 675.658012] Lustre: 25148:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 675.663162] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 675.668806] Lustre: 25148:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 725.474830] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 725.481651] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 725.498755] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 728.356330] Lustre: server umount lustre-OST07c6 complete [ 737.431622] Lustre: server umount lustre-MDT0000 complete [ 750.874166] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 753.680513] Key type lgssc unregistered [ 753.950717] LNet: 28718:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.953920] LNetError: 28718:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.962585] LNet: Removed LNI 192.168.202.128@tcp [ 754.756338] Key type .llcrypt unregistered [ 754.758474] Key type ._llcrypt unregistered [ 772.359117] Key type ._llcrypt registered [ 772.360498] Key type .llcrypt registered [ 772.463792] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 773.756550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 773.900851] alg: No test for adler32 (adler32-zlib) [ 774.977385] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 775.212555] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 776.896283] Key type lgssc registered [ 777.937445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 791.566546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 791.642460] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 793.877597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 800.934754] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 803.301343] LustreError: 29989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 803.320118] LustreError: 29989:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 808.420415] LustreError: 29990:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 813.561251] LustreError: 29989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 814.513427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 815.478414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 824.218340] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 839.017320] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 849.534917] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 859.126975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 859.711209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 863.794996] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 867.096755] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 876.177518] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 893.842971] Lustre: server umount lustre-OST0000 complete [ 899.553502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 899.555067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 899.562752] Lustre: Skipped 1 previous similar message [ 904.674199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 904.679876] Lustre: Skipped 1 previous similar message [ 908.870470] Lustre: server umount lustre-MDT0000 complete [ 909.795102] LustreError: 29985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 909.810312] LustreError: 29985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 910.816110] Lustre: 29300:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485798/real 1782485798] req@00000000d9c864af x1869071702332032/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782485814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 910.851897] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 912.676136] Lustre: 29299:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782485800/real 1782485800] req@00000000d791c3cd x1869071702332544/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782485816 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 913.216498] LustreError: 29969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485816 with bad export cookie 4225683813402472703 [ 913.219862] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 913.229587] LustreError: 29969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 913.580223] Lustre: server umount lustre-MDT0001 complete [ 925.985590] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 929.197031] Key type lgssc unregistered [ 929.542842] LNet: 33732:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 929.554849] LNetError: 33732:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 930.602164] LNet: Removed LNI 192.168.202.128@tcp [ 931.341856] Key type .llcrypt unregistered [ 931.343818] Key type ._llcrypt unregistered [ 951.658760] Key type ._llcrypt registered [ 951.660304] Key type .llcrypt registered [ 951.754331] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 952.999103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 953.157139] alg: No test for adler32 (adler32-zlib) [ 954.424678] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 954.639398] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 956.393247] Key type lgssc registered [ 957.679499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 965.657239] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 965.674431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 967.334192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.542373] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 977.889322] LustreError: 35000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 977.906982] LustreError: 35000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 982.480837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 983.048969] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 988.606631] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 998.030313] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1003.478666] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1010.964369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1011.537932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1012.616164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1018.510042] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1026.955930] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1038.309127] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1038.328620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1038.329580] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1038.344059] Lustre: Skipped 1 previous similar message [ 1043.433337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1043.446132] Lustre: Skipped 1 previous similar message [ 1044.414267] Lustre: server umount lustre-OST0000 complete [ 1048.560452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1048.589037] Lustre: Skipped 1 previous similar message [ 1048.594975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.666062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1053.673560] Lustre: Skipped 2 previous similar messages [ 1058.786511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1058.802767] Lustre: Skipped 1 previous similar message [ 1062.370985] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1062.549387] Lustre: server umount lustre-MDT0000 complete [ 1063.905497] LustreError: 35000:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1063.922844] LustreError: 35000:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1066.497209] LustreError: 36847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782485970 with bad export cookie 1756428617662702397 [ 1066.498968] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1066.512678] LustreError: 36847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1066.861839] Lustre: server umount lustre-MDT0001 complete [ 1079.102511] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1082.603186] Key type lgssc unregistered [ 1083.022525] LNet: 38539:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1083.030802] LNetError: 38539:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1084.076573] LNet: Removed LNI 192.168.202.128@tcp [ 1085.052826] Key type .llcrypt unregistered [ 1085.057216] Key type ._llcrypt unregistered [ 1111.417173] Key type ._llcrypt registered [ 1111.419308] Key type .llcrypt registered [ 1111.545222] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1113.294243] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1113.324075] alg: No test for adler32 (adler32-zlib) [ 1114.443041] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1114.728519] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1116.457273] Key type lgssc registered [ 1118.066881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1126.037199] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1126.053854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.545466] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1133.146152] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1138.152617] LustreError: 39956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.185155] LustreError: 39956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1143.271681] LustreError: 39957:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1144.452447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.102824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1151.025680] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1162.571224] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1170.704277] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1179.441254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1180.050345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1187.882865] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1196.853750] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1197.500237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1204.111725] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1206.253081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1206.265178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1206.280180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.785360] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.790750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.839223] Lustre: Skipped 2 previous similar messages [ 1209.605438] Lustre: server umount lustre-MDT0000 complete [ 1212.909887] LustreError: 39956:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.934091] LustreError: 39956:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1213.901091] LustreError: 39938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486117 with bad export cookie 13370136719679324316 [ 1213.912229] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1213.915928] LustreError: 39938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1214.344299] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [ 1214.349457] Lustre: Skipped 1 previous similar message [ 1218.024903] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1218.028820] LustreError: 39953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1218.050775] Lustre: Skipped 1 previous similar message [ 1218.073883] LustreError: 39953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1219.425445] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [ 1219.439294] Lustre: Skipped 1 previous similar message [ 1223.137626] LustreError: 39953:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1223.149349] LustreError: 39953:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1228.256337] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1228.478137] Lustre: server umount lustre-MDT0001 complete [ 1243.379513] Lustre: server umount lustre-OST0000 complete [ 1260.906837] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1263.549955] Key type lgssc unregistered [ 1263.899729] LNet: 43832:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1263.914370] LNetError: 43832:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1264.945268] LNet: Removed LNI 192.168.202.128@tcp [ 1265.645063] Key type .llcrypt unregistered [ 1265.647200] Key type ._llcrypt unregistered [ 1286.366641] Key type ._llcrypt registered [ 1286.370912] Key type .llcrypt registered [ 1286.485577] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1287.596069] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1287.617322] alg: No test for adler32 (adler32-zlib) [ 1288.785468] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1289.035148] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1290.680224] Key type lgssc registered [ 1291.903861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.317996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.330779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1299.680831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1303.639045] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1310.194839] LustreError: 45228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1310.216322] LustreError: 45228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1311.263331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1311.533934] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1315.363412] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1322.169944] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1326.667061] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1331.915711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1332.181703] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.500685] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1343.056989] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1344.505154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1353.615912] Lustre: lustre-OST0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1354.723613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1354.723896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1354.731874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1354.742133] Lustre: Skipped 1 previous similar message [ 1354.752708] Lustre: Skipped 1 previous similar message [ 1358.606929] Lustre: server umount lustre-OST0000 complete [ 1378.282083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1378.286821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1378.311658] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.322317] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1380.325179] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.975187] Lustre: server umount lustre-MDT0000 complete [ 1384.947131] LustreError: 45209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486288 with bad export cookie 1152604635318516272 [ 1384.955512] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1384.955965] LustreError: 45209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1399.264209] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1399.553853] Lustre: server umount lustre-MDT0001 complete [ 1410.150446] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1412.456597] Key type lgssc unregistered [ 1412.690962] LNet: 48937:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.699652] LNetError: 48937:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1412.711974] LNet: Removed LNI 192.168.202.128@tcp [ 1413.407234] Key type .llcrypt unregistered [ 1413.409246] Key type ._llcrypt unregistered [ 1433.090565] Key type ._llcrypt registered [ 1433.092804] Key type .llcrypt registered [ 1433.211238] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1434.239844] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1434.290480] alg: No test for adler32 (adler32-zlib) [ 1435.417910] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1435.643272] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1437.283981] Key type lgssc registered [ 1438.158227] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1444.411311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1444.426105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1445.844189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1449.760772] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1456.101816] LustreError: 50333:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1456.111417] LustreError: 50333:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1457.810208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.201833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1462.034131] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1468.352529] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1472.780570] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1477.557631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1477.906657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1481.540303] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1485.780910] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1491.261801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1495.326994] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1497.789345] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1509.358181] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-2611h06m52s [ 1510.272849] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1511.914088] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1511.920031] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1511.930114] Lustre: Skipped 1 previous similar message [ 1511.946374] Lustre: Skipped 2 previous similar messages [ 1512.834343] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1513.901954] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 1513.919540] Lustre: Skipped 3 previous similar messages [ 1517.685425] Lustre: server umount lustre-MDT0000 complete [ 1518.962613] LustreError: 51392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1520.381238] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1525.096384] LustreError: 51392:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.122872] LustreError: 51392:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1525.912190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1526.039626] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1526.242476] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1526.326398] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1527.289549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1530.075328] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1532.763532] Lustre: DEBUG MARKER: Wait for df (36543) ... [ 1534.094072] Lustre: DEBUG MARKER: done [ 1537.509234] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1537.532387] Lustre: Skipped 1 previous similar message [ 1537.540472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1537.547508] Lustre: Skipped 1 previous similar message [ 1542.629047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1542.644870] Lustre: Skipped 1 previous similar message [ 1542.966327] Lustre: server umount lustre-OST0000 complete [ 1547.746630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.750128] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1547.753868] Lustre: Skipped 1 previous similar message [ 1552.644333] Lustre: server umount lustre-MDT0000 complete [ 1553.833611] LustreError: 51061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1553.851534] LustreError: 51061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 1556.177078] LustreError: 50313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486459 with bad export cookie 8114383049242741862 [ 1556.184554] LustreError: 50313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1556.186915] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1556.496236] Lustre: server umount lustre-MDT0001 complete [ 1565.725900] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1567.844640] Key type lgssc unregistered [ 1568.081376] LNet: 55254:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1568.085736] LNetError: 55254:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1568.097628] LNet: Removed LNI 192.168.202.128@tcp [ 1568.679841] Key type .llcrypt unregistered [ 1568.681885] Key type ._llcrypt unregistered [ 1597.277746] Key type ._llcrypt registered [ 1597.279658] Key type .llcrypt registered [ 1597.362622] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1598.088089] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1598.119238] alg: No test for adler32 (adler32-zlib) [ 1599.107358] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1599.311218] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1600.937126] Key type lgssc registered [ 1601.731678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1607.638119] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1607.652311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1608.834440] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1608.848505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1608.953624] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1611.920634] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1618.746228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1618.808317] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1618.830372] Lustre: 57443:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1619.038862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1622.285331] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1627.382264] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1631.112857] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1636.874534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1637.008155] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1637.180222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1641.763260] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1643.651627] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1647.670544] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1652.663783] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-2611h09m16s [ 1653.727964] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1659.064617] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1661.044298] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1661.089996] LustreError: 57450:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1681.277668] Lustre: lustre-MDT0000: Client 591f307c-dd74-488a-870d-60bcf94cf301 (at 192.168.202.28@tcp) reconnecting [ 1701.177040] LustreError: 57450:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1702.883858] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1704.671395] Lustre: DEBUG MARKER: done [ 1710.049178] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1710.065139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.078518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1710.565856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1710.566053] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.580032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.776308] Lustre: server umount lustre-OST0000 complete [ 1720.804984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1720.806827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1720.816896] Lustre: Skipped 1 previous similar message [ 1723.404868] Lustre: server umount lustre-MDT0000 complete [ 1726.348185] LustreError: 59313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486630 with bad export cookie 4379749864104855996 [ 1726.349441] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1726.357479] LustreError: 59313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1726.599284] Lustre: server umount lustre-MDT0001 complete [ 1735.140449] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 1737.018633] Key type lgssc unregistered [ 1737.296525] LNet: 61766:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1737.307575] LNetError: 61766:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1737.320788] LNet: Removed LNI 192.168.202.128@tcp [ 1737.879240] Key type .llcrypt unregistered [ 1737.880617] Key type ._llcrypt unregistered [ 1780.547501] Key type ._llcrypt registered [ 1780.550666] Key type .llcrypt registered [ 1780.691638] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 1781.675584] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1781.716512] alg: No test for adler32 (adler32-zlib) [ 1783.265715] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 1783.597688] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 1785.280218] Key type lgssc registered [ 1786.814085] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1793.823600] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1793.904720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1795.475379] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1795.490728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1795.576685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1797.059123] Lustre: Failing over lustre-MDT0000 [ 1797.580776] Lustre: server umount lustre-MDT0000 complete [ 1805.807851] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:11:48 (1782486708) [ 1814.253849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1815.124383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1819.783391] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1828.021881] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1828.091943] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1828.114878] Lustre: 64566:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1828.281738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1832.106786] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1838.694967] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1844.674476] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1850.623486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1850.796621] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1851.045144] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1856.685210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1857.437232] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1864.072424] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1870.367440] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1877.475195] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1877.479143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1877.488881] Lustre: Skipped 1 previous similar message [ 1877.686629] Lustre: server umount lustre-MDT0000 complete [ 1881.034066] LustreError: 64558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486784 with bad export cookie 6108726736379698005 [ 1881.046302] LustreError: 64558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1881.049290] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1881.483395] Lustre: server umount lustre-MDT0001 complete [ 1884.581684] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1889.223322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1897.662507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1898.976201] Lustre: 63489:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782486786/real 1782486786] req@000000003c8b0263 x1869072760345728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782486802 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1899.012089] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1899.023131] Lustre: Skipped 1 previous similar message [ 1903.008107] Lustre: 63490:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782486790/real 1782486790] req@00000000d46c1d12 x1869072760345856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782486806 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1904.032151] Lustre: 63491:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782486791/real 1782486791] req@0000000050a53167 x1869072760345984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782486807 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1906.214472] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x54c68e99efd7e43f [ 1906.219529] Lustre: MGC192.168.202.128@tcp: Connection restored to 0@lo (at 0@lo) [ 1906.552249] LustreError: 66415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1906.559190] LustreError: 66415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1906.633036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1906.725868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1907.617226] Lustre: 63489:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782486794/real 1782486794] req@00000000c2dbfc96 x1869072760346240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782486810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1908.710808] LustreError: 68337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.731929] LustreError: 68337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1911.175675] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1911.777361] Lustre: 63489:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782486799/real 1782486799] req@00000000d8bc9bd8 x1869072760346368/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782486815 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1913.826393] LustreError: 68336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1913.851362] LustreError: 68336:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1917.925304] LustreError: 68336:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1917.927865] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1919.021676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1919.299940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1922.849369] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1924.588721] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1929.073865] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1934.218678] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1939.939757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1939.943589] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1939.957707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1939.967662] Lustre: Skipped 2 previous similar messages [ 1941.936805] Lustre: server umount lustre-MDT0000 complete [ 1944.729630] LustreError: 64558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486848 with bad export cookie 6108726736379700287 [ 1944.737392] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1944.737617] LustreError: 64558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1945.059807] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1945.071603] Lustre: Skipped 2 previous similar messages [ 1945.074784] LustreError: 68337:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.095785] LustreError: 68337:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1945.264974] Lustre: server umount lustre-MDT0001 complete [ 1950.627162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1959.424175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1969.569691] LustreError: 63487:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fbaae624 x1869072760370944/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1969.848849] LustreError: 66416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1969.915636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1969.973183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1973.727861] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1980.973416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1983.277138] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1985.010499] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 1990.997199] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1993.514356] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1997.633312] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2003.955850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.960936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2003.969462] Lustre: Skipped 2 previous similar messages [ 2003.973494] Lustre: Skipped 1 previous similar message [ 2005.030845] Lustre: server umount lustre-MDT0000 complete [ 2008.168112] LustreError: 64559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782486911 with bad export cookie 6108726736379701666 [ 2008.182467] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2008.430756] Lustre: server umount lustre-MDT0001 complete [ 2014.783913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2018.055728] Lustre: DEBUG MARKER: files compared the same [ 2029.964915] Lustre: server umount lustre-OST0000 complete [ 2044.654593] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2046.979670] Key type lgssc unregistered [ 2047.203869] LNet: 74289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2047.213978] LNetError: 74289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2047.233571] LNet: Removed LNI 192.168.202.128@tcp [ 2047.854624] Key type .llcrypt unregistered [ 2047.856698] Key type ._llcrypt unregistered [ 2068.127787] Key type ._llcrypt registered [ 2068.129835] Key type .llcrypt registered [ 2068.220970] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2069.134926] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2069.212173] alg: No test for adler32 (adler32-zlib) [ 2070.262954] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2070.433576] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2072.096189] Key type lgssc registered [ 2072.898377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2078.725665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2078.738040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2080.030438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2083.532156] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug +malloc all [ 2089.447230] LustreError: 75684:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.460614] LustreError: 75684:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2090.413517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2090.619428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2094.084526] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug +malloc all [ 2100.310755] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2104.891230] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2110.845905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2111.109215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2116.236818] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug +malloc all [ 2122.294513] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2124.405195] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2135.010020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2135.010072] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2135.018199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2135.023547] Lustre: Skipped 1 previous similar message [ 2135.031244] Lustre: Skipped 1 previous similar message [ 2139.505744] Lustre: server umount lustre-OST0000 complete [ 2142.178068] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2142.183270] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.192686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.249125] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2145.258634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2145.262881] Lustre: Skipped 1 previous similar message [ 2148.329823] Lustre: server umount lustre-MDT0000 complete [ 2150.369630] LustreError: 75683:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2150.393643] LustreError: 75683:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2151.678597] LustreError: 75663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487055 with bad export cookie 11352915426185701955 [ 2151.686141] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2151.694983] LustreError: 75663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2151.976630] Lustre: server umount lustre-MDT0001 complete [ 2161.188245] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2163.601830] Key type lgssc unregistered [ 2163.826336] LNet: 79215:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2163.834072] LNetError: 79215:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2163.849388] LNet: Removed LNI 192.168.202.128@tcp [ 2164.464734] Key type .llcrypt unregistered [ 2164.466860] Key type ._llcrypt unregistered [ 2182.711201] Key type ._llcrypt registered [ 2182.712959] Key type .llcrypt registered [ 2182.793946] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2183.309908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2183.326159] alg: No test for adler32 (adler32-zlib) [ 2184.337356] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2184.516348] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2186.148395] Key type lgssc registered [ 2187.116754] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2192.709483] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2192.728615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2208.288237] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2213.408793] LustreError: 80583:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [ 2240.101902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.127607] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2249.069241] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2253.819223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.900197] Lustre: *** cfs_fail_loc=706, val=0*** [ 2254.127424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2254.181649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2257.350550] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2260.455188] LustreError: 81669:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2260.470132] LustreError: 81669:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 2263.749432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2263.970302] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2266.943426] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2271.966402] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2275.672281] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2279.400159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2279.401596] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.413218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2279.417959] Lustre: Skipped 1 previous similar message [ 2283.185120] Lustre: server umount lustre-OST0000 complete [ 2289.635232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.636515] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2289.637541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2289.637546] Lustre: Skipped 1 previous similar message [ 2289.643043] Lustre: Skipped 1 previous similar message [ 2292.010865] Lustre: server umount lustre-MDT0000 complete [ 2294.344344] LustreError: 80589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487198 with bad export cookie 417323092027750300 [ 2294.344818] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2294.349124] LustreError: 80589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2294.564649] Lustre: server umount lustre-MDT0001 complete [ 2300.667644] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2302.346159] Key type lgssc unregistered [ 2302.523816] LNet: 84194:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2302.531257] LNetError: 84194:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2303.591862] LNet: Removed LNI 192.168.202.128@tcp [ 2304.058157] Key type .llcrypt unregistered [ 2304.059965] Key type ._llcrypt unregistered [ 2319.051140] Key type ._llcrypt registered [ 2319.052448] Key type .llcrypt registered [ 2319.116513] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2319.785516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2319.799099] alg: No test for adler32 (adler32-zlib) [ 2320.744197] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2320.859878] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2322.472171] Key type lgssc registered [ 2323.004153] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2327.770242] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2327.786765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.575450] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2336.036689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2336.333992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2338.750188] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2341.347564] LustreError: 86274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.357953] LustreError: 86274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2343.703273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2343.917903] LustreError: 86302:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.937350] LustreError: 86302:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2343.948803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.043605] LustreError: 86277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.049429] LustreError: 86277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2347.621206] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2352.098472] LustreError: 86274:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.116721] LustreError: 86274:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2352.754548] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2352.764537] Lustre: MGS: Client 6fc6108d-8d6c-4877-a556-c390a197683b (at 0@lo) reconnecting [ 2352.774832] LustreError: 86288:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2352.915472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.962508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2355.823617] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2368.481403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2368.488337] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.501065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2368.504943] Lustre: Skipped 1 previous similar message [ 2371.955862] Lustre: server umount lustre-OST0000 complete [ 2378.724393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2378.730392] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.734421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2378.734430] Lustre: Skipped 1 previous similar message [ 2378.756079] Lustre: Skipped 1 previous similar message [ 2380.238742] Lustre: server umount lustre-MDT0000 complete [ 2382.511532] Lustre: server umount lustre-MDT0001 complete [ 2384.542134] Lustre: server umount MGS complete [ 2391.011526] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:21:34 (1782487294) [ 2406.954864] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 2412.140672] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2431.729276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2434.056928] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2439.519441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2439.662072] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2439.689302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2439.736899] Lustre: lustre-MDT0001: new disk, initializing [ 2439.779060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.799514] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2439.806063] Lustre: 93438:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2439.816440] LustreError: 93438:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2442.086834] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2448.201834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2448.367161] Lustre: lustre-OST0000: new disk, initializing [ 2448.370528] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2448.374307] Lustre: 94203:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2448.414873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.650139] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2456.606422] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2457.120705] Lustre: 94865:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000ae79c6ce x1869073323430016/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782487376 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2459.969208] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2459.975948] Lustre: MGS: Client 1b2ac799-9996-4894-a6f8-5f3d87875a37 (at 0@lo) reconnecting [ 2459.989773] LustreError: 93424:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2460.014702] Lustre: 92707:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2460.049395] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2460.098588] Lustre: lustre-MDT0000: new disk, initializing [ 2460.175709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.186452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2461.218599] Lustre: 95274:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000047e0420d x1869073323435520/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1782487380 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2461.280692] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2462.067715] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2462.183678] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2462.193116] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2463.269920] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2463.275455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2463.340970] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2482.657162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2482.659876] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2482.665469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2482.665476] Lustre: Skipped 1 previous similar message [ 2482.675587] Lustre: Skipped 1 previous similar message [ 2487.780448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.783825] Lustre: Skipped 2 previous similar messages [ 2488.100270] Lustre: server umount lustre-OST0000 complete [ 2492.898597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2492.906339] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2492.917058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.922530] Lustre: Skipped 1 previous similar message [ 2496.165571] Lustre: server umount lustre-MDT0000 complete [ 2498.020123] LustreError: 93409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.031074] LustreError: 93409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2504.449195] Lustre: server umount lustre-MDT0001 complete [ 2506.039135] Lustre: server umount MGS complete [ 2511.369632] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:23:35 (1782487415) [ 2519.200742] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2520.692870] Key type lgssc unregistered [ 2520.849830] LNet: 97763:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2520.858249] LNetError: 97763:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2520.865698] LNet: Removed LNI 192.168.202.128@tcp [ 2521.319430] Key type .llcrypt unregistered [ 2521.324321] Key type ._llcrypt unregistered [ 2530.091394] Key type ._llcrypt registered [ 2530.093243] Key type .llcrypt registered [ 2530.147101] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2530.557574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2530.609404] alg: No test for adler32 (adler32-zlib) [ 2531.542985] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2531.658154] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2533.256246] Key type lgssc registered [ 2533.759257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2537.657608] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2537.669626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2538.928164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2541.159959] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2545.823994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.022386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2548.139865] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2551.684457] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2554.656421] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2558.159544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2558.340765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.201397] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2563.501521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2564.808393] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2578.913630] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2578.922051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2578.931077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2581.986831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2581.989048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.990984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2584.633312] Lustre: server umount lustre-OST0000 complete [ 2587.108679] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2587.109224] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2587.114807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2587.120621] Lustre: Skipped 1 previous similar message [ 2587.123979] Lustre: Skipped 1 previous similar message [ 2592.228878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2592.236458] Lustre: Skipped 1 previous similar message [ 2592.808151] Lustre: server umount lustre-MDT0000 complete [ 2595.138387] LustreError: 98943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487498 with bad export cookie 17967351475532517248 [ 2595.141619] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2595.144046] LustreError: 98943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2595.318180] Lustre: server umount lustre-MDT0001 complete [ 2601.008162] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2602.309914] Key type lgssc unregistered [ 2602.459435] LNet: 102507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2602.462924] LNetError: 102507:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2602.475387] LNet: Removed LNI 192.168.202.128@tcp [ 2602.858911] Key type .llcrypt unregistered [ 2602.860675] Key type ._llcrypt unregistered [ 2611.321705] Key type ._llcrypt registered [ 2611.323176] Key type .llcrypt registered [ 2611.390930] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2612.029273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2612.040197] alg: No test for adler32 (adler32-zlib) [ 2612.955537] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2613.067846] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2614.672148] Key type lgssc registered [ 2615.281531] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2618.778332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2618.779620] LustreError: 103567:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2618.781915] Lustre: Skipped 1 previous similar message [ 2618.793664] LustreError: 103568:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2618.797034] LustreError: 103567:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4991 [ 2618.809082] LustreError: 103567:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2618.817109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2618.818413] LustreError: 103567:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2620.142560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2624.794770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.990372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2626.969523] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2629.524417] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lsmod [ 2630.992947] LustreError: 104773:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 2631.003697] LustreError: 104775:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 2631.007508] LustreError: 104773:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4987 [ 2631.011615] LustreError: 104773:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2631.015197] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2631.017530] LustreError: 104773:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2631.232766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.787572] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.791803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2634.801944] Lustre: Skipped 1 previous similar message [ 2639.841991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2639.846509] Lustre: Skipped 4 previous similar messages [ 2644.963700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2644.967355] Lustre: Skipped 2 previous similar messages [ 2647.008126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2647.089617] Lustre: server umount lustre-MDT0000 complete [ 2648.862947] LustreError: 103585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487552 with bad export cookie 15652028995476713426 [ 2648.864578] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2648.870912] LustreError: 103585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2649.144245] LustreError: 104036:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000f349690 x1869073629595904/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2649.219141] Lustre: server umount lustre-MDT0001 complete [ 2661.586702] Lustre: server umount lustre-OST0000 complete [ 2666.192632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2666.385834] LustreError: 105839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2666.391116] LustreError: 105839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2666.420740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.082738] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2671.586581] LustreError: 105840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2672.187801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.386734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2674.478364] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2677.883385] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2680.359214] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2683.175026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.309541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2685.611542] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2688.437913] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2691.495369] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2702.820772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.821328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2702.822355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2702.826493] Lustre: Skipped 2 previous similar messages [ 2703.899851] Lustre: server umount lustre-OST0000 complete [ 2705.848577] Lustre: server umount lustre-MDT0000 complete [ 2707.580872] LustreError: 105819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487611 with bad export cookie 15652028995476714112 [ 2707.584462] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2707.586412] LustreError: 105819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2713.793195] Lustre: server umount lustre-MDT0001 complete [ 2718.067747] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2719.288757] Key type lgssc unregistered [ 2719.441768] LNet: 109387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2719.444698] LNetError: 109387:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2719.455068] LNet: Removed LNI 192.168.202.128@tcp [ 2719.862515] Key type .llcrypt unregistered [ 2719.864113] Key type ._llcrypt unregistered [ 2730.245760] Key type ._llcrypt registered [ 2730.247038] Key type .llcrypt registered [ 2730.291732] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2730.672098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2730.706876] alg: No test for adler32 (adler32-zlib) [ 2731.598496] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2731.689337] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2733.280165] Key type lgssc registered [ 2733.688900] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2736.536031] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2736.543610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2737.723435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2739.388531] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2742.883734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.090683] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2744.667590] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2747.653653] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2750.060310] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2752.714067] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2752.873855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.158226] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2757.990032] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2763.115428] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 2770.459831] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2771.422385] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2773.475629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2773.475901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2773.478050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2773.483398] Lustre: Skipped 1 previous similar message [ 2774.818889] Lustre: server umount lustre-OST0000 complete [ 2778.595372] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2778.610432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2778.613932] Lustre: Skipped 2 previous similar messages [ 2778.735107] Lustre: server umount lustre-MDT0000 complete [ 2780.690050] LustreError: 113331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487684 with bad export cookie 6132537398071195223 [ 2780.691124] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2780.697316] LustreError: 113331:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2780.869250] Lustre: server umount lustre-MDT0001 complete [ 2785.750427] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2787.056530] Key type lgssc unregistered [ 2787.191354] LNet: 114433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2787.194093] LNetError: 114433:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2787.203669] LNet: Removed LNI 192.168.202.128@tcp [ 2787.620446] Key type .llcrypt unregistered [ 2787.622162] Key type ._llcrypt unregistered [ 2795.786592] Key type ._llcrypt registered [ 2795.787784] Key type .llcrypt registered [ 2795.843457] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2796.340741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2796.347912] alg: No test for adler32 (adler32-zlib) [ 2797.234415] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2797.363025] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2798.961874] Key type lgssc registered [ 2799.481873] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2802.998318] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2803.009028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2804.219538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2805.972683] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2809.939962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2810.154557] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2812.490481] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2816.230340] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2818.865907] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2822.130809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2822.310085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2824.852230] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2827.721797] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2828.456311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 2844.131024] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.140185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.162261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2845.160996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2845.165119] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2845.184244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2848.720395] Lustre: server umount lustre-OST0000 complete [ 2862.104938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.125428] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.127617] Lustre: Skipped 1 previous similar message [ 2863.318587] Lustre: server umount lustre-MDT0000 complete [ 2865.635910] LustreError: 116753:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2865.710915] LustreError: 116753:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2869.412363] LustreError: 115612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487773 with bad export cookie 17122163145368993496 [ 2869.422840] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2869.783549] Lustre: server umount lustre-MDT0001 complete [ 2888.477338] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 2892.717366] Key type lgssc unregistered [ 2893.126962] LNet: 119207:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2893.149964] LNetError: 119207:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2893.165224] LNet: Removed LNI 192.168.202.128@tcp [ 2894.237428] Key type .llcrypt unregistered [ 2894.239455] Key type ._llcrypt unregistered [ 2931.426330] Key type ._llcrypt registered [ 2931.429277] Key type .llcrypt registered [ 2931.601365] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 2933.133369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2933.404637] alg: No test for adler32 (adler32-zlib) [ 2934.541323] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 2934.905653] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 2936.696164] Key type lgssc registered [ 2938.000761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2949.317302] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2949.357871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2951.181639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.378375] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2960.428269] LustreError: 120629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2960.458126] LustreError: 120629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2965.480777] LustreError: 120630:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2966.415535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2966.958103] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2972.076417] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 2980.835303] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2988.151509] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2995.872270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2996.480263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3003.136184] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3011.151186] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3021.050597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3027.979743] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3032.642722] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3032.663158] Lustre: Skipped 2 previous similar messages [ 3036.242948] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3036.272868] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3037.496448] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3037.504981] Lustre: Skipped 2 previous similar messages [ 3045.559250] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3045.573569] Lustre: Skipped 1 previous similar message [ 3050.741662] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.28@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3050.747221] Lustre: Skipped 2 previous similar messages [ 3055.819655] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.28@tcp 192.168.0.[2,10]@tcp [ 3062.352151] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.28/24@tcp in log lustre-MDT0000 [ 3062.357090] Lustre: Skipped 2 previous similar messages [ 3065.313346] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.28/24@tcp [ 3065.328590] Lustre: Skipped 1 previous similar message [ 3071.988376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3072.001060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3072.004190] LustreError: Skipped 2 previous similar messages [ 3072.051868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3077.544878] Lustre: server umount lustre-OST0000 complete [ 3082.726053] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.730142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3082.759936] Lustre: Skipped 2 previous similar messages [ 3082.773739] Lustre: Skipped 3 previous similar messages [ 3087.856475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3087.870640] Lustre: Skipped 1 previous similar message [ 3088.187676] Lustre: server umount lustre-MDT0000 complete [ 3092.403239] LustreError: 123283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782487996 with bad export cookie 8369425327062756818 [ 3092.411405] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3092.417515] LustreError: 123283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3092.972812] Lustre: server umount lustre-MDT0001 complete [ 3104.672648] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3107.699926] Key type lgssc unregistered [ 3107.982727] LNet: 125549:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3107.992882] LNetError: 125549:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3108.043970] LNet: Removed LNI 192.168.202.128@tcp [ 3108.959229] Key type .llcrypt unregistered [ 3108.962185] Key type ._llcrypt unregistered [ 3137.841584] Key type ._llcrypt registered [ 3137.857601] Key type .llcrypt registered [ 3137.987310] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3139.492276] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3139.529172] alg: No test for adler32 (adler32-zlib) [ 3140.616885] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3140.956884] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3142.648186] Key type lgssc registered [ 3143.834437] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3151.842867] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3151.865078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.531716] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3153.572042] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.28/24@tcp [ 3153.573160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3158.920402] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3164.129489] LustreError: 127124:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3164.179644] LustreError: 127124:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3169.250202] LustreError: 127123:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3170.217822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3171.164342] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3171.239752] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.28/24@tcp [ 3171.331325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.224037] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3189.912601] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3196.392268] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3204.595727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3205.365978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.473541] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3221.319901] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3223.834588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3234.280153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3234.291498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3234.291866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3234.296021] Lustre: Skipped 1 previous similar message [ 3234.309708] Lustre: Skipped 1 previous similar message [ 3239.404664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3240.555632] Lustre: server umount lustre-OST0000 complete [ 3248.099287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3248.106493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3248.132383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3248.141779] Lustre: Skipped 1 previous similar message [ 3249.636506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3251.801113] Lustre: server umount lustre-MDT0000 complete [ 3254.755916] LustreError: 127119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.779652] LustreError: 127119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3255.607232] LustreError: 127105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488159 with bad export cookie 10622731087089830550 [ 3255.618499] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3255.632704] LustreError: 127105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3256.040933] Lustre: server umount lustre-MDT0001 complete [ 3268.776661] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3272.074232] Key type lgssc unregistered [ 3272.369757] LNet: 130714:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3272.393121] LNetError: 130714:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3273.472289] LNet: Removed LNI 192.168.202.128@tcp [ 3274.935335] Key type .llcrypt unregistered [ 3274.937273] Key type ._llcrypt unregistered [ 3308.281846] Key type ._llcrypt registered [ 3308.288211] Key type .llcrypt registered [ 3308.439033] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3325.682628] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3327.352478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3327.414945] alg: No test for adler32 (adler32-zlib) [ 3328.839867] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3329.102744] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3330.736222] Key type lgssc registered [ 3332.075795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3370.636969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3370.673322] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3371.974753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3372.010622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3372.128776] Lustre: lustre-MDT0000: new disk, initializing [ 3372.259469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.285442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3377.964662] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3391.397810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3391.560520] Lustre: 134118:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3391.619881] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3391.627341] Lustre: Skipped 1 previous similar message [ 3391.707358] Lustre: lustre-MDT0001: new disk, initializing [ 3391.834778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3391.892914] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3391.920381] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3396.597984] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3405.735398] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3411.203439] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3418.109646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3418.503673] Lustre: lustre-OST0000: new disk, initializing [ 3418.507107] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3418.516987] Lustre: 136085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3418.643927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.835396] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3426.941509] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3426.955580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3427.127449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3434.756923] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3438.654733] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3438.849772] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3443.276813] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3443.428101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3447.779260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3447.782641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3447.789026] LustreError: Skipped 1 previous similar message [ 3447.801958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3447.818409] Lustre: Skipped 1 previous similar message [ 3451.425475] Lustre: server umount lustre-OST0000 complete [ 3458.039955] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3458.047213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3458.069415] Lustre: Skipped 2 previous similar messages [ 3458.092654] Lustre: Skipped 1 previous similar message [ 3461.696102] Lustre: server umount lustre-MDT0000 complete [ 3467.935144] LustreError: 134110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488371 with bad export cookie 11022600396658559198 [ 3467.947508] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3467.952817] LustreError: 134110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3468.467317] Lustre: server umount lustre-MDT0001 complete [ 3483.119482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3484.177019] LustreError: 138303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3484.200677] LustreError: 138303:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3484.316587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3489.763735] LustreError: 138304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3492.207739] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3493.861053] LustreError: 138303:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3498.977852] LustreError: 138304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3502.400377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3502.635036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3508.775298] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3516.500273] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3523.177945] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3535.090458] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3536.204495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3547.369149] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3556.574726] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3563.875839] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3564.024761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3568.457475] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3568.732565] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3577.305102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3577.450766] Lustre: lustre-OST0001: new disk, initializing [ 3577.460807] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3577.468068] Lustre: 141676:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3577.567180] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3583.086569] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3585.598685] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3585.616182] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3585.682619] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3590.692711] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3590.871923] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3596.148190] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3596.568208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3601.844278] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff960c8b32c000.ost_server_uuid 50 [ 3602.923630] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff960c8b32c000.ost_server_uuid in FULL state after 0 sec [ 3611.125442] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.130043] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3611.132277] Lustre: Skipped 1 previous similar message [ 3611.145849] Lustre: Skipped 1 previous similar message [ 3616.229095] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3616.240958] Lustre: Skipped 1 previous similar message [ 3621.347853] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3621.355525] Lustre: Skipped 1 previous similar message [ 3622.368225] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3622.451137] Lustre: server umount lustre-OST0001 complete [ 3624.930087] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3624.939152] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3625.954390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3626.488599] LustreError: 140125:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3626.502974] LustreError: 140125:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3631.588090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3631.592889] Lustre: Skipped 5 previous similar messages [ 3636.708492] LustreError: 140127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3636.721067] LustreError: 140127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3639.264143] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3639.436903] Lustre: server umount lustre-OST0000 complete [ 3645.921871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3645.925817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3645.930788] Lustre: Skipped 1 previous similar message [ 3648.354375] Lustre: server umount lustre-MDT0000 complete [ 3650.500195] LustreError: 139038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488554 with bad export cookie 11022600396658560080 [ 3650.503518] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3650.507158] LustreError: 139038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3650.737646] Lustre: server umount lustre-MDT0001 complete [ 3662.795861] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3664.505888] Key type lgssc unregistered [ 3664.687714] LNet: 144680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3664.691595] LNetError: 144680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3664.712526] LNet: Removed LNI 192.168.202.128@tcp [ 3665.272881] Key type .llcrypt unregistered [ 3665.275052] Key type ._llcrypt unregistered [ 3682.604320] Key type ._llcrypt registered [ 3682.605759] Key type .llcrypt registered [ 3682.641669] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3688.757585] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3689.187158] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3689.194180] alg: No test for adler32 (adler32-zlib) [ 3690.049078] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3690.131852] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3691.722804] Key type lgssc registered [ 3692.116589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3703.584993] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3703.590982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3704.679849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3704.690912] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3704.725435] Lustre: lustre-MDT0000: new disk, initializing [ 3704.750391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3704.756794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3706.052491] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3710.068825] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3710.112717] Lustre: 148803:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3710.133402] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3710.135014] Lustre: Skipped 1 previous similar message [ 3710.168267] Lustre: lustre-MDT0001: new disk, initializing [ 3710.200893] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3710.211903] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3710.215154] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3711.477749] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3715.047439] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3716.859736] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3719.081748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3719.175997] Lustre: lustre-OST0000: new disk, initializing [ 3719.178195] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3719.180802] Lustre: 150771:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3719.209860] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3721.079268] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3721.910876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3721.914499] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3721.952095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3724.875721] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3726.821091] Lustre: Failing over lustre-OST0000 [ 3726.875482] Lustre: server umount lustre-OST0000 complete [ 3739.668074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3739.765642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3739.770206] Lustre: Skipped 1 previous similar message [ 3739.777681] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3739.784081] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3741.536242] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3741.607182] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3742.857635] Lustre: Failing over lustre-MDT0000 [ 3742.956677] Lustre: server umount lustre-MDT0000 complete [ 3743.904129] Lustre: 147035:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782488631/real 1782488631] req@00000000c183b2a7 x1869074758919040/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782488647 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3745.249695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.254843] Lustre: Skipped 2 previous similar messages [ 3748.320194] Lustre: 147034:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782488636/real 1782488636] req@00000000bc117577 x1869074758920832/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782488652 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3748.328421] Lustre: 147034:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3750.369174] LustreError: 148815:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3750.375520] LustreError: 148815:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3751.768414] LustreError: 148810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3754.081088] Lustre: 147035:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782488641/real 1782488641] req@00000000d1127299 x1869074758921984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782488657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3755.489148] LustreError: 148816:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3755.496540] LustreError: 148816:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3755.817824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3755.860164] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3755.864456] Lustre: 148808:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.202.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3755.929977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 3755.948051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3755.967454] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3756.888584] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3757.297093] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3761.127211] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3761.133897] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 3761.155272] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3766.244570] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3766.244825] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3766.246817] LustreError: Skipped 1 previous similar message [ 3766.250936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3766.253561] Lustre: Skipped 1 previous similar message [ 3768.329647] Lustre: server umount lustre-OST0000 complete [ 3771.362054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3771.363081] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.365432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3771.368902] Lustre: Skipped 2 previous similar messages [ 3771.373831] Lustre: Skipped 1 previous similar message [ 3775.642089] Lustre: server umount lustre-MDT0000 complete [ 3776.480918] LustreError: 149617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3776.485932] LustreError: 149617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3776.899112] LustreError: 149600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488680 with bad export cookie 7143923706846376654 [ 3776.901557] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3776.904017] LustreError: 149600:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3777.035693] Lustre: server umount lustre-MDT0001 complete [ 3780.653550] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 3781.699747] Key type lgssc unregistered [ 3781.832353] LNet: 154417:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3781.835768] LNetError: 154417:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3781.844476] LNet: Removed LNI 192.168.202.128@tcp [ 3782.145730] Key type .llcrypt unregistered [ 3782.147025] Key type ._llcrypt unregistered [ 3802.878020] Key type ._llcrypt registered [ 3802.879812] Key type .llcrypt registered [ 3802.921390] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3803.297166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3803.333814] alg: No test for adler32 (adler32-zlib) [ 3804.196436] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 3804.281061] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 3805.864145] Key type lgssc registered [ 3806.245394] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3808.854864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3808.862324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3810.017900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3811.440245] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3815.118783] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3817.174602] LustreError: 155965:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3817.179344] LustreError: 155965:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3817.182761] LustreError: 155965:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 3817.301499] Lustre: server umount lustre-MDT0000 complete [ 3822.368087] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 3825.573716] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 3838.695810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3838.799874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3838.816245] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3838.857904] Lustre: lustre-MDT0000: new disk, initializing [ 3838.916455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3838.931883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3842.129945] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3850.492813] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3850.550413] Lustre: 159260:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3850.583109] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3850.587299] Lustre: Skipped 1 previous similar message [ 3850.637912] Lustre: lustre-MDT0001: new disk, initializing [ 3850.740023] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3850.776098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3850.783252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3854.163331] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3861.265659] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3865.394450] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3870.719085] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3870.919328] Lustre: lustre-OST0000: new disk, initializing [ 3870.922267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3870.928220] Lustre: 161227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3870.992533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3877.644258] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 3879.493962] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3879.506509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3879.653592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3890.043418] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3894.940923] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3898.428471] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:46:40 (1782488800) [ 4076.005513] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4076.014210] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4076.044211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.584808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.585043] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4079.623060] Lustre: Skipped 1 previous similar message [ 4081.467906] Lustre: server umount lustre-MDT0000 complete [ 4083.978283] LustreError: 159251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782488987 with bad export cookie 6344801564875088654 [ 4083.981644] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4083.986436] LustreError: 159251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4084.704688] LustreError: 161992:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4084.704708] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4084.723845] LustreError: 161992:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4084.748500] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4084.751242] Lustre: Skipped 2 previous similar messages [ 4088.736549] LustreError: 159268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4088.738108] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4090.283495] Lustre: server umount lustre-MDT0001 complete [ 4107.232367] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4107.299603] Lustre: server umount lustre-OST0000 complete [ 4114.001178] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4120.752908] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4146.761761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.958796] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4146.981361] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4147.057968] Lustre: lustre-MDT0000: new disk, initializing [ 4147.165640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.186486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4151.133417] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4160.816597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4160.914499] Lustre: 165875:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4160.971567] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4160.983132] Lustre: Skipped 1 previous similar message [ 4161.115213] Lustre: lustre-MDT0001: new disk, initializing [ 4161.223218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4161.260912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4161.268218] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4165.014236] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4171.024661] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4174.982877] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4178.420387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4178.576491] Lustre: lustre-OST0000: new disk, initializing [ 4178.580974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4178.586594] Lustre: 167853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4178.645480] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4183.415347] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4185.178427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4185.191132] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4185.366500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4199.157666] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4207.184073] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4207.678245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4215.197511] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4215.547125] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4221.410102] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4221.429844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4221.436150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4221.468337] Lustre: Skipped 1 previous similar message [ 4221.494877] Lustre: Skipped 1 previous similar message [ 4225.680154] Lustre: server umount lustre-OST0000 complete [ 4233.185720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4233.193280] LustreError: Skipped 1 previous similar message [ 4233.197739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4233.216574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4239.806992] Lustre: server umount lustre-MDT0000 complete [ 4241.890149] LustreError: 167108:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4241.910863] LustreError: 167108:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4244.356835] LustreError: 165866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489148 with bad export cookie 6344801564875247491 [ 4244.369451] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4244.380748] LustreError: 165866:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4244.943964] Lustre: server umount lustre-MDT0001 complete [ 4254.919232] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:52:37 (1782489157) [ 4272.646594] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4281.887489] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4324.941885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4325.213700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4325.235181] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4325.297118] Lustre: lustre-MDT0000: new disk, initializing [ 4325.403974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4325.428141] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4329.943903] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4341.789585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.881834] Lustre: 172612:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4342.234397] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4347.440686] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4357.359927] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4364.060450] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4372.673039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4373.216302] Lustre: lustre-OST0000: new disk, initializing [ 4373.219535] Lustre: Skipped 1 previous similar message [ 4373.221266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4373.226073] Lustre: Skipped 2 previous similar messages [ 4373.228735] Lustre: 174590:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4380.206870] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4381.736339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4381.745934] Lustre: Skipped 1 previous similar message [ 4381.750799] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4381.880676] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4389.708325] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4402.657240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4402.658343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4402.663024] LustreError: Skipped 1 previous similar message [ 4402.668332] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4402.690024] Lustre: Skipped 2 previous similar messages [ 4402.723360] Lustre: Skipped 3 previous similar messages [ 4405.258449] Lustre: server umount lustre-OST0000 complete [ 4415.359842] Lustre: server umount lustre-MDT0000 complete [ 4418.022511] LustreError: 172625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4418.048516] LustreError: 172625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4420.231354] LustreError: 172604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489323 with bad export cookie 6344801564875248373 [ 4420.255573] LustreError: 172604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4420.275495] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4420.684230] Lustre: server umount lustre-MDT0001 complete [ 4430.662222] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:55:33 (1782489333) [ 4447.883243] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 4457.856673] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4495.329214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4495.611251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4495.633751] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4495.730705] Lustre: lustre-MDT0000: new disk, initializing [ 4495.810789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.816578] Lustre: Skipped 2 previous similar messages [ 4495.827675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4501.209199] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4512.064605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4512.175388] Lustre: 178912:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4512.433842] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4517.555978] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4526.344549] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4531.446174] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4537.938727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4538.237454] Lustre: 180891:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4543.697583] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4544.270399] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4553.585947] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4564.455639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4564.455712] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4564.468151] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4564.468163] Lustre: Skipped 2 previous similar messages [ 4564.471410] Lustre: Skipped 3 previous similar messages [ 4567.551824] Lustre: server umount lustre-OST0000 complete [ 4573.671293] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4573.686713] LustreError: Skipped 1 previous similar message [ 4578.447148] Lustre: server umount lustre-MDT0000 complete [ 4579.811766] LustreError: 178918:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4579.841885] LustreError: 178918:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4582.990131] LustreError: 178905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489486 with bad export cookie 6344801564875249458 [ 4583.000703] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4583.015788] LustreError: 155246:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting [ 4583.354852] Lustre: server umount lustre-MDT0001 complete [ 4595.606966] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4598.730821] Key type lgssc unregistered [ 4598.988391] LNet: 182748:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4598.994331] LNetError: 182748:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4599.005995] LNet: Removed LNI 192.168.202.128@tcp [ 4599.933302] Key type .llcrypt unregistered [ 4599.935241] Key type ._llcrypt unregistered [ 4624.547374] Key type ._llcrypt registered [ 4624.551459] Key type .llcrypt registered [ 4624.700189] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4626.021761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4626.301902] alg: No test for adler32 (adler32-zlib) [ 4627.448541] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 4627.742782] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4629.432145] Key type lgssc registered [ 4630.976421] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4640.600535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4640.644105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.165580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4646.776699] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4651.505728] LustreError: 184165:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.536491] LustreError: 184165:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4655.492655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4655.950896] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4660.823297] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4668.893891] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4674.387608] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4681.445122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4681.847982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4686.971310] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4688.760988] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4696.045364] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4721.137215] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4721.144979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4721.146124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4721.160737] Lustre: Skipped 1 previous similar message [ 4724.349526] Lustre: server umount lustre-OST0000 complete [ 4731.366685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4731.377873] Lustre: Skipped 1 previous similar message [ 4731.383315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4731.394410] Lustre: Skipped 2 previous similar messages [ 4736.482419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4736.495954] Lustre: Skipped 1 previous similar message [ 4740.579083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4740.584949] Lustre: Skipped 1 previous similar message [ 4743.136202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4743.245330] Lustre: server umount lustre-MDT0000 complete [ 4745.699603] LustreError: 184166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4745.740498] LustreError: 184166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4747.618576] LustreError: 184147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489651 with bad export cookie 15882635281730386477 [ 4747.621579] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4747.634138] LustreError: 184147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4748.166777] Lustre: server umount lustre-MDT0001 complete [ 4761.952745] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 4765.368347] Key type lgssc unregistered [ 4765.813410] LNet: 187735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4765.820487] LNetError: 187735:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4765.858203] LNet: Removed LNI 192.168.202.128@tcp [ 4766.684875] Key type .llcrypt unregistered [ 4766.687438] Key type ._llcrypt unregistered [ 4791.867275] Key type ._llcrypt registered [ 4791.872065] Key type .llcrypt registered [ 4792.035916] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 4793.924023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4794.115816] alg: No test for adler32 (adler32-zlib) [ 4795.417173] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 4795.920879] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 4797.648864] Key type lgssc registered [ 4799.546588] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4810.601036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4810.610393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.023659] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.242898] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4821.486553] LustreError: 189158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4821.521741] LustreError: 189158:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4826.599600] LustreError: 189159:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4831.599595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4831.735332] LustreError: 189158:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4832.264314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4836.635320] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4843.613494] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4849.398340] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4855.266265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4855.525927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4856.564623] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4860.969194] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4868.476113] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4876.156983] Lustre: lustre-OST0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 4877.281941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4877.287947] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4877.296875] LustreError: Skipped 1 previous similar message [ 4877.299939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4877.308549] Lustre: Skipped 1 previous similar message [ 4881.018863] Lustre: server umount lustre-OST0000 complete [ 4889.006607] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff960c88a85800.ost_server_uuid 50 [ 4890.698724] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4903.905684] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4903.910347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4903.917605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4903.919824] Lustre: Skipped 1 previous similar message [ 4908.012032] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4908.022872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4908.041497] Lustre: Skipped 1 previous similar message [ 4910.068323] Lustre: server umount lustre-MDT0000 complete [ 4913.134389] LustreError: 189887:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.155586] LustreError: 189887:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4914.904419] LustreError: 189139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489818 with bad export cookie 246163885308763554 [ 4914.913707] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4914.917985] LustreError: 189139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4915.531656] Lustre: server umount lustre-MDT0001 complete [ 4924.816623] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:03:47 (1782489827) [ 4931.523299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4931.873716] LustreError: 193078:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4931.972426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.234266] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4941.286250] LustreError: 193079:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4941.310472] LustreError: 193079:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4946.735497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4947.633956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4951.737703] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4960.420454] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4965.663212] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4972.320529] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4972.454034] LustreError: 194897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.476904] LustreError: 194897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4972.579349] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4976.705765] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4978.424588] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 4984.896145] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4990.550699] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4990.772626] Lustre: lustre-OST0001: new disk, initializing [ 4990.776182] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4990.780861] Lustre: 195923:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4990.875890] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4996.701102] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5000.731235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5000.743034] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5000.783970] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5005.255343] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5010.920288] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5010.925296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5010.941639] Lustre: Skipped 1 previous similar message [ 5010.948418] Lustre: Skipped 1 previous similar message [ 5021.162859] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5021.168192] Lustre: Skipped 5 previous similar messages [ 5022.176268] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5022.274220] Lustre: server umount lustre-OST0000 complete [ 5026.286235] LustreError: 194897:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.300472] LustreError: 194897:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5027.703757] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5027.894218] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5032.268867] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5032.432279] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5046.754571] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5046.763132] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5046.775918] LustreError: Skipped 1 previous similar message [ 5046.787382] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5046.799420] Lustre: Skipped 1 previous similar message [ 5046.829519] Lustre: Skipped 2 previous similar messages [ 5052.711577] Lustre: server umount lustre-OST0001 complete [ 5056.993263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5057.031439] Lustre: Skipped 1 previous similar message [ 5063.384595] Lustre: server umount lustre-MDT0000 complete [ 5066.979267] LustreError: 193059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782489970 with bad export cookie 246163885308764625 [ 5066.990510] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5066.991561] LustreError: 193059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5067.256265] Lustre: server umount lustre-MDT0001 complete [ 5087.564320] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:06:31 (1782489991) [ 5092.715145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5092.975099] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5093.022118] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5093.266591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5098.182677] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5107.355182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5107.543211] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5107.599917] Lustre: 199360:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5112.930390] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5121.266575] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5127.078805] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5134.124272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5134.264191] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5139.644525] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5140.094775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5147.764935] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5154.700275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5154.802961] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5161.909035] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5168.953711] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5173.248791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5173.269377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5173.275488] Lustre: Skipped 11 previous similar messages [ 5177.402358] Lustre: server umount lustre-OST0000 complete [ 5178.339047] LustreError: 201208:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5178.371450] LustreError: 201208:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 5193.187800] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5198.257601] Lustre: server umount lustre-OST0001 complete [ 5208.482948] Lustre: server umount lustre-MDT0000 complete [ 5212.066388] LustreError: 199350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490115 with bad export cookie 246163885308766032 [ 5212.076946] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5212.082620] LustreError: 199350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 5212.489555] Lustre: server umount lustre-MDT0001 complete [ 5236.299155] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:08:59 (1782490139) [ 5250.069061] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 5257.864894] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 5290.416981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.607576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5290.626568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5290.682517] Lustre: lustre-MDT0000: new disk, initializing [ 5290.747206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.751104] Lustre: Skipped 3 previous similar messages [ 5290.763386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5294.806845] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5304.647547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.756579] Lustre: 207454:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5304.800326] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5304.803667] Lustre: Skipped 1 previous similar message [ 5304.867492] Lustre: lustre-MDT0001: new disk, initializing [ 5305.002115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5305.018072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5309.334239] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5318.657653] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5323.914328] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5330.667194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5330.962715] Lustre: lustre-OST0000: new disk, initializing [ 5330.970612] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5330.980527] Lustre: 209417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5335.734277] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5341.290889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5341.311293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5341.503185] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5344.269946] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5348.477165] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5348.654492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5352.777517] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5353.008893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5357.027569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5357.029569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.046942] LustreError: Skipped 2 previous similar messages [ 5357.047777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5357.047781] Lustre: Skipped 5 previous similar messages [ 5357.091063] Lustre: Skipped 6 previous similar messages [ 5369.312711] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5369.537338] Lustre: server umount lustre-OST0000 complete [ 5376.994970] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5387.232326] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5387.523254] Lustre: server umount lustre-MDT0000 complete [ 5387.746333] LustreError: 207465:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5387.757871] LustreError: 207465:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 5390.861902] LustreError: 207445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490294 with bad export cookie 246163885308767593 [ 5390.870525] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5390.877227] LustreError: 207445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5391.288344] Lustre: server umount lustre-MDT0001 complete [ 5401.454288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.928533] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5417.787967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5425.136398] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5436.395273] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5443.968686] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5451.174707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5451.550158] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5451.556634] Lustre: Skipped 4 previous similar messages [ 5459.584336] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5471.507365] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5479.326683] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5479.770739] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5487.200833] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5488.004416] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5492.715527] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5492.720704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5492.731625] Lustre: Skipped 3 previous similar messages [ 5492.742892] Lustre: Skipped 11 previous similar messages [ 5498.008125] Lustre: server umount lustre-OST0000 complete [ 5505.958511] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5510.392887] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5538.292933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5544.567676] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5547.890149] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.905154] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5547.927217] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5551.539096] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5558.329956] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5561.565683] Lustre: server umount lustre-OST0000 complete [ 5566.947285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5579.234182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5579.539137] Lustre: server umount lustre-MDT0000 complete [ 5579.745086] Lustre: 188468:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782490467/real 1782490467] req@00000000db8030a2 x1869075918876032/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782490483 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5582.925961] LustreError: 211619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490486 with bad export cookie 246163885308768475 [ 5582.930026] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5582.934244] LustreError: 211619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5583.294621] Lustre: server umount lustre-MDT0001 complete [ 5592.075929] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:14:55 (1782490495) [ 5599.247855] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5605.314371] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5613.981989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5618.464973] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5625.260199] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5630.396593] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5637.482339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5646.132326] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5653.108869] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5658.204302] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5658.366748] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5663.119338] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5663.429599] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5671.065647] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5671.237901] Lustre: lustre-OST0001: new disk, initializing [ 5671.240787] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5671.245147] Lustre: 220983:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5678.090634] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5679.230453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5679.238339] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5679.321709] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5688.379526] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5693.390459] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5693.566988] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5697.928874] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5698.101562] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5700.070606] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5700.078385] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5700.095352] Lustre: Skipped 4 previous similar messages [ 5714.400351] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5714.595827] Lustre: server umount lustre-OST0001 complete [ 5715.453377] LustreError: 219436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5715.504619] LustreError: 219436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 5723.101234] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5723.409245] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5730.219023] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5730.335932] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5758.212746] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5758.382992] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5758.389208] Lustre: Skipped 5 previous similar messages [ 5763.586143] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5763.598392] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5763.606449] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5763.614631] Lustre: Skipped 1 previous similar message [ 5764.755812] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5771.904720] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5773.353506] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5776.773693] Lustre: lustre-OST0001: Not available for connect from 192.168.202.28@tcp (stopping) [ 5776.786407] Lustre: Skipped 13 previous similar messages [ 5778.917308] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5778.922946] LustreError: Skipped 2 previous similar messages [ 5781.504063] Lustre: server umount lustre-OST0001 complete [ 5814.241167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5814.483932] Lustre: server umount lustre-OST0000 complete [ 5824.789388] Lustre: server umount lustre-MDT0000 complete [ 5828.984846] LustreError: 217595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490732 with bad export cookie 246163885308769441 [ 5829.003717] LustreError: 217595:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5829.010234] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5835.433607] Lustre: server umount lustre-MDT0001 complete [ 5868.177526] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:19:30 (1782490770) [ 5875.581519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5875.932788] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5875.968393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5881.006907] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5889.090278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5889.225471] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5889.269053] Lustre: 226232:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5894.036400] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5900.508446] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5906.794937] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5915.225916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5915.381447] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5922.239260] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5930.920400] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5941.227848] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5941.341882] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5950.556832] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 5960.448493] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5961.961807] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5966.653743] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5967.172696] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5972.411380] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5972.687639] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5979.951698] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff960c8b1f4000.ost_server_uuid 50 [ 5982.001441] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff960c8b1f4000.ost_server_uuid in IDLE state after 0 sec [ 5983.262931] Lustre: Permanently deactivating lustre-OST0001 [ 5983.279729] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5988.289259] Lustre: Permanently reactivating lustre-OST0001 [ 5988.308461] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5988.313503] Lustre: Skipped 2 previous similar messages [ 5993.968437] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5993.973963] LustreError: Skipped 3 previous similar messages [ 5993.984547] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.996287] Lustre: Skipped 6 previous similar messages [ 5998.742995] Lustre: server umount lustre-OST0001 complete [ 6009.920250] Lustre: server umount lustre-OST0000 complete [ 6021.890795] Lustre: server umount lustre-MDT0000 complete [ 6024.740161] LustreError: 226224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782490928 with bad export cookie 246163885308770729 [ 6024.743867] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6024.751385] LustreError: 226224:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6025.047351] Lustre: server umount lustre-MDT0001 complete [ 6048.377722] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:22:31 (1782490951) [ 6057.481145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6058.071376] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6058.108819] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6058.127636] Lustre: Skipped 2 previous similar messages [ 6064.935242] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6077.080834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6077.265782] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6077.293232] Lustre: 232928:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6083.835811] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6093.574905] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6102.630895] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6111.454931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6111.775096] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6111.826036] Lustre: Permanently deactivating lustre-OST0000 [ 6111.837913] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6118.118330] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6120.501213] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6126.672581] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6135.173544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6135.276874] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6143.605474] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6144.455984] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6152.292246] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6153.957284] Lustre: Permanently reactivating lustre-OST0000 [ 6154.005659] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6154.016405] Lustre: Skipped 2 previous similar messages [ 6163.484625] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6163.486728] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 43 seconds. I think it's dead, and I am evicting it. exp 0000000047c8bf5c, cur 1782491067 deadline 1782491054 last 1782491024 [ 6163.488983] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6163.490893] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6163.523299] Lustre: Skipped 1 previous similar message [ 6163.541970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6172.505761] Lustre: server umount lustre-OST0001 complete [ 6182.832566] Lustre: server umount lustre-OST0000 complete [ 6201.312951] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6201.656847] Lustre: server umount lustre-MDT0000 complete [ 6206.555161] LustreError: 232920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491110 with bad export cookie 246163885308772395 [ 6206.568040] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6206.574636] LustreError: 232920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6207.036826] Lustre: server umount lustre-MDT0001 complete [ 6219.961652] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6222.965827] Key type lgssc unregistered [ 6223.319626] LNet: 237810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6223.331667] LNetError: 237810:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6223.346074] LNet: Removed LNI 192.168.202.128@tcp [ 6224.405266] Key type .llcrypt unregistered [ 6224.410061] Key type ._llcrypt unregistered [ 6248.428339] Key type ._llcrypt registered [ 6248.430560] Key type .llcrypt registered [ 6248.523045] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6249.865312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6249.915493] alg: No test for adler32 (adler32-zlib) [ 6251.072040] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 6251.375456] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6253.064370] Key type lgssc registered [ 6254.299291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6265.007240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6265.045751] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6266.680123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6271.823874] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6277.096355] LustreError: 239277:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6277.132777] LustreError: 239277:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6281.765606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6281.943787] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6281.996235] Lustre: 239265:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6282.018611] Lustre: Permanently deactivating lustre-MDT0001 [ 6282.028438] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6282.258510] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6286.291735] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6289.975400] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6294.123222] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6301.053698] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6308.872691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6309.386619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6311.404544] LustreError: 241119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.446143] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6316.524026] LustreError: 241255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.954722] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6321.638892] LustreError: 241119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.773196] LustreError: 241121:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6326.793460] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6326.794387] LustreError: 241121:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6327.934731] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6335.992603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6336.274823] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6341.631081] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6341.662529] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6344.019797] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6352.652270] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6356.494287] Lustre: Permanently reactivating lustre-MDT0001 [ 6356.523848] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6356.528858] Lustre: Skipped 2 previous similar messages [ 6361.479798] Lustre: lustre-MDT0001: Client cbb5aad3-b704-4612-aeb2-8fd23f6122a4 (at 192.168.202.28@tcp) reconnecting [ 6363.342056] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6363.361107] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6363.368802] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6363.383399] LustreError: 242844:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6363.394677] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6363.398074] LustreError: 242844:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6363.398225] LustreError: 242844:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.202.128@tcp: cfg command failed: rc = -108 [ 6363.398621] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6363.398621] [ 6363.411482] LustreError: 239269:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6369.505560] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6371.480684] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6377.657430] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6377.986506] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6383.228838] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6383.420857] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6385.936867] Lustre: Permanently deactivating lustre-MDT0001 [ 6385.949224] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6385.954338] Lustre: Skipped 2 previous similar messages [ 6394.981606] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6405.093151] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6405.095525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6405.117918] Lustre: Skipped 2 previous similar messages [ 6405.121674] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6405.136164] Lustre: Skipped 3 previous similar messages [ 6410.048852] Lustre: server umount lustre-MDT0000 complete [ 6410.227345] LustreError: 243719:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6410.261532] LustreError: 243719:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6414.536483] LustreError: 239257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491318 with bad export cookie 10796944403194616867 [ 6414.538757] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6414.546854] LustreError: 239257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6414.756973] Lustre: server umount lustre-MDT0001 complete [ 6421.688769] Lustre: server umount lustre-OST0000 complete [ 6433.970419] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:28:56 (1782491336) [ 6445.724987] Lustre: server umount lustre-OST0001 complete [ 6452.868295] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 6463.799064] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6499.736434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6499.951972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6499.958881] Lustre: Skipped 2 previous similar messages [ 6499.983824] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6500.050825] Lustre: lustre-MDT0000: new disk, initializing [ 6500.131911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6500.155877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6504.554910] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6514.953674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6515.068441] Lustre: 247562:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6515.115420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6515.119962] Lustre: Skipped 1 previous similar message [ 6515.182674] Lustre: lustre-MDT0001: new disk, initializing [ 6515.255476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6515.308132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6515.316316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6519.284387] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6527.779581] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6532.911038] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6538.544796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6538.792993] Lustre: lustre-OST0000: new disk, initializing [ 6538.802170] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6538.806470] Lustre: 249530:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6538.883083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6543.994229] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6547.076366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6547.087633] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6547.194950] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6552.288630] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6554.310474] LustreError: 247568:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6562.051773] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6562.215546] Lustre: lustre-OST0001: new disk, initializing [ 6562.219639] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6562.226880] Lustre: 250643:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6564.384529] LustreError: 247568:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6564.409193] LustreError: 247570:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6567.910440] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6569.021102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6569.033078] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6569.173670] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6574.418888] LustreError: 247570:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6577.073599] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6579.065029] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6584.817687] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6584.826862] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6584.828774] LustreError: Skipped 1 previous similar message [ 6584.832028] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6584.857966] Lustre: Skipped 2 previous similar messages [ 6584.872084] Lustre: Skipped 1 previous similar message [ 6587.039404] Lustre: server umount lustre-OST0001 complete [ 6589.927106] LustreError: 250651:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6589.953761] LustreError: 250651:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6603.747969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6603.753921] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6603.762871] LustreError: Skipped 1 previous similar message [ 6603.768604] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6603.784497] Lustre: Skipped 1 previous similar message [ 6603.805709] Lustre: Skipped 2 previous similar messages [ 6609.532850] Lustre: server umount lustre-OST0000 complete [ 6615.524356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6615.528085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6615.546451] Lustre: Skipped 2 previous similar messages [ 6615.547883] Lustre: Skipped 1 previous similar message [ 6619.023660] Lustre: server umount lustre-MDT0000 complete [ 6621.944208] LustreError: 247553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491525 with bad export cookie 10796944403194619674 [ 6621.944965] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6621.953924] LustreError: 247553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6628.369829] Lustre: server umount lustre-MDT0001 complete [ 6638.540559] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6641.232701] Key type lgssc unregistered [ 6641.501215] LNet: 252676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6641.507325] LNetError: 252676:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6641.529950] LNet: Removed LNI 192.168.202.128@tcp [ 6642.321365] Key type .llcrypt unregistered [ 6642.323797] Key type ._llcrypt unregistered [ 6679.405691] Key type ._llcrypt registered [ 6679.411579] Key type .llcrypt registered [ 6679.510654] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6680.716044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6680.746628] alg: No test for adler32 (adler32-zlib) [ 6681.782638] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 6682.000860] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6683.664189] Key type lgssc registered [ 6684.860905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6691.888947] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6691.907833] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6693.178614] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6693.194693] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6693.292221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6697.678123] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6705.205160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6705.293910] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6705.323803] Lustre: 254885:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6705.545530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6709.181440] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6714.537481] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6718.762360] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6723.830908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.947906] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6724.119787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6729.245985] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6732.440762] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6735.783334] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6783.457344] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6783.467712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.479691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6783.973134] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6783.990063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6783.995323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6787.717572] Lustre: server umount lustre-OST0000 complete [ 6791.110587] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6800.119497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6800.344552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6803.437551] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6803.452270] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6803.456046] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6803.476237] Lustre: Skipped 1 previous similar message [ 6805.195370] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6810.278164] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6849.508579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6849.516099] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6849.517416] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6849.528369] Lustre: Skipped 1 previous similar message [ 6853.024898] Lustre: server umount lustre-OST0000 complete [ 6859.238857] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6859.251553] LustreError: Skipped 1 previous similar message [ 6859.261946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6859.273605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6859.275710] Lustre: Skipped 1 previous similar message [ 6862.628983] Lustre: server umount lustre-MDT0000 complete [ 6866.071929] LustreError: 254877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491769 with bad export cookie 8996924071773826361 [ 6866.080123] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6866.083724] LustreError: 254877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6866.432904] Lustre: server umount lustre-MDT0001 complete [ 6874.452662] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6876.383581] Key type lgssc unregistered [ 6876.672531] LNet: 260310:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6876.681305] LNetError: 260310:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6876.695267] LNet: Removed LNI 192.168.202.128@tcp [ 6877.326651] Key type .llcrypt unregistered [ 6877.331358] Key type ._llcrypt unregistered [ 6895.100882] Key type ._llcrypt registered [ 6895.105674] Key type .llcrypt registered [ 6895.170957] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 6896.001973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6896.028560] alg: No test for adler32 (adler32-zlib) [ 6897.090738] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 6897.278268] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 6898.944218] Key type lgssc registered [ 6899.849693] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6905.819194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6905.831534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6907.139375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6910.623225] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6916.587129] LustreError: 261706:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6916.600343] LustreError: 261706:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6917.810202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6918.158393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6921.739966] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6927.352906] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6931.563136] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6936.683096] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6936.950533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6941.249505] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 6946.380506] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6951.280508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6968.801264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6968.806616] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6968.822392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6971.876650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6971.876843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.882717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.891979] Lustre: Skipped 1 previous similar message [ 6974.181751] Lustre: server umount lustre-OST0000 complete [ 6977.003112] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6977.011786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6977.016037] Lustre: Skipped 1 previous similar message [ 6977.027916] Lustre: Skipped 1 previous similar message [ 6982.113574] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6982.117964] Lustre: Skipped 1 previous similar message [ 6982.940274] Lustre: server umount lustre-MDT0000 complete [ 6985.459724] LustreError: 261686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491889 with bad export cookie 15496874602606084601 [ 6985.469746] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6985.470296] LustreError: 261686:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6985.730179] Lustre: server umount lustre-MDT0001 complete [ 6992.939919] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 6994.988715] Key type lgssc unregistered [ 6995.200799] LNet: 266009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6995.205635] LNetError: 266009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6996.265508] LNet: Removed LNI 192.168.202.128@tcp [ 6996.848984] Key type .llcrypt unregistered [ 6996.851890] Key type ._llcrypt unregistered [ 7009.754472] Key type ._llcrypt registered [ 7009.756325] Key type .llcrypt registered [ 7009.840414] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7010.576145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7010.698915] alg: No test for adler32 (adler32-zlib) [ 7011.674354] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 7011.853336] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7013.456201] Key type lgssc registered [ 7014.357309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7020.699991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7020.723457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7022.043347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7025.352472] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7030.937391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7031.125594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7033.973330] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7039.035399] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7042.626978] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7048.005543] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7048.413362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7053.351979] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7059.067294] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7066.862063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7077.347788] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7077.354910] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7077.375447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7082.468732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7082.471977] Lustre: Skipped 2 previous similar messages [ 7083.036983] Lustre: server umount lustre-OST0000 complete [ 7087.591197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7087.591838] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7087.593411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7087.609637] Lustre: Skipped 2 previous similar messages [ 7091.879694] Lustre: server umount lustre-MDT0000 complete [ 7092.710365] LustreError: 267938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7092.720066] LustreError: 267938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7094.805663] LustreError: 267192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782491998 with bad export cookie 7346423846391961939 [ 7094.816894] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7094.819798] LustreError: 267192:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7095.092755] Lustre: server umount lustre-MDT0001 complete [ 7102.721250] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7104.862333] Key type lgssc unregistered [ 7105.148497] LNet: 270905:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7105.155726] LNetError: 270905:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7105.182403] LNet: Removed LNI 192.168.202.128@tcp [ 7105.816315] Key type .llcrypt unregistered [ 7105.818958] Key type ._llcrypt unregistered [ 7123.724366] Key type ._llcrypt registered [ 7123.726653] Key type .llcrypt registered [ 7123.819780] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7124.589257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7124.639111] alg: No test for adler32 (adler32-zlib) [ 7125.638640] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 7125.823487] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7127.464272] Key type lgssc registered [ 7128.195775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7133.503625] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7133.519106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.913531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7137.864186] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7143.887240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7144.134987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7147.546696] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7152.929757] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7157.132815] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7161.804852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7162.019270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7166.954559] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7173.033395] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7179.439152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7200.232917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7200.240331] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7200.242706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7200.245774] Lustre: Skipped 1 previous similar message [ 7201.987675] Lustre: server umount lustre-OST0000 complete [ 7205.347736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7205.352879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7205.355262] Lustre: Skipped 2 previous similar messages [ 7205.367143] Lustre: Skipped 1 previous similar message [ 7210.468831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7210.479521] Lustre: Skipped 1 previous similar message [ 7211.310898] Lustre: server umount lustre-MDT0000 complete [ 7213.847132] LustreError: 274145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492117 with bad export cookie 11923946401154213901 [ 7213.850577] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7213.855698] LustreError: 274145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7214.064781] Lustre: server umount lustre-MDT0001 complete [ 7221.595541] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7223.401677] Key type lgssc unregistered [ 7223.605635] LNet: 276607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7223.609386] LNetError: 276607:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7223.622862] LNet: Removed LNI 192.168.202.128@tcp [ 7224.304783] Key type .llcrypt unregistered [ 7224.307098] Key type ._llcrypt unregistered [ 7236.345726] Key type ._llcrypt registered [ 7236.348356] Key type .llcrypt registered [ 7236.413573] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7237.050870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7237.157434] alg: No test for adler32 (adler32-zlib) [ 7238.126499] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 7238.279206] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7239.912141] Key type lgssc registered [ 7240.764952] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7245.963563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7246.002896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7247.298079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7250.054781] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7255.238950] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7255.472199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7258.228825] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7263.874479] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7267.824418] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7272.525854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7272.705180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7273.712910] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7276.870949] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7281.708771] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7289.314615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7289.319922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7289.327273] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7291.366796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7291.368179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7291.371391] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7295.263937] Lustre: server umount lustre-OST0000 complete [ 7301.601560] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7301.606048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.616160] Lustre: Skipped 1 previous similar message [ 7301.619564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7301.624941] Lustre: Skipped 2 previous similar messages [ 7306.725298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7306.731494] Lustre: Skipped 1 previous similar message [ 7311.841550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7311.845720] Lustre: Skipped 1 previous similar message [ 7312.352124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7312.583242] Lustre: server umount lustre-MDT0000 complete [ 7315.630920] LustreError: 277788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492219 with bad export cookie 16377625748307457057 [ 7315.640965] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7315.644033] LustreError: 277788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7316.053737] Lustre: server umount lustre-MDT0001 complete [ 7323.557287] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7325.496736] Key type lgssc unregistered [ 7325.720801] LNet: 281522:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7325.725393] LNetError: 281522:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7325.746665] LNet: Removed LNI 192.168.202.128@tcp [ 7326.301407] Key type .llcrypt unregistered [ 7326.303400] Key type ._llcrypt unregistered [ 7333.744789] Key type ._llcrypt registered [ 7333.746853] Key type .llcrypt registered [ 7333.826827] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7345.652886] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 7356.509411] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7357.158952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7357.182826] alg: No test for adler32 (adler32-zlib) [ 7358.144976] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 7358.274925] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7359.896207] Key type lgssc registered [ 7360.594112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7384.587578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7384.615636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7385.877981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7385.930177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7386.021261] Lustre: lustre-MDT0000: new disk, initializing [ 7386.099810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7386.128351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7389.351443] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7397.896601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7397.990334] Lustre: 285048:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7398.045964] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7398.048979] Lustre: Skipped 1 previous similar message [ 7398.111572] Lustre: lustre-MDT0001: new disk, initializing [ 7398.200651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7398.263292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7398.278208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7401.912949] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7409.152183] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7413.431981] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7418.425451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7418.566961] Lustre: lustre-OST0000: new disk, initializing [ 7418.568820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7418.572228] Lustre: 287019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7418.620662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7423.005734] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7427.633578] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7427.645314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7427.710756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7429.701925] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7432.810311] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7432.948284] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7436.115823] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7436.275567] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7438.308613] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7438.312450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7438.324295] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7438.340700] Lustre: Skipped 1 previous similar message [ 7443.428714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7443.437696] Lustre: Skipped 1 previous similar message [ 7443.842917] Lustre: server umount lustre-OST0000 complete [ 7448.547736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.548427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7448.560485] Lustre: Skipped 2 previous similar messages [ 7448.571018] Lustre: Skipped 1 previous similar message [ 7452.640425] Lustre: server umount lustre-MDT0000 complete [ 7455.165929] LustreError: 285041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492358 with bad export cookie 7162020470944681480 [ 7455.168044] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7455.179078] LustreError: 285041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7455.360194] Lustre: server umount lustre-MDT0001 complete [ 7460.634163] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:46:04 (1782492364) [ 7465.395336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7465.661143] LustreError: 289428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.676776] LustreError: 289428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7465.721042] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7468.432535] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7471.074098] LustreError: 289429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7474.124535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.412109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7477.536430] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7482.325363] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7486.492947] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7491.557920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7491.802388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7496.204677] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7501.179644] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7520.737772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7520.742780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7520.754190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7523.583170] Lustre: server umount lustre-OST0000 complete [ 7526.881539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.900156] Lustre: Skipped 1 previous similar message [ 7526.904234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7526.908064] Lustre: Skipped 3 previous similar messages [ 7532.414612] Lustre: server umount lustre-MDT0000 complete [ 7535.103755] LustreError: 289408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492438 with bad export cookie 7162020470944682362 [ 7535.110627] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7535.121257] LustreError: 289408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7535.398567] Lustre: server umount lustre-MDT0001 complete [ 7545.087623] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 7548.234293] Key type lgssc unregistered [ 7548.649157] LNet: 292982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7548.659025] LNetError: 292982:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7548.685699] LNet: Removed LNI 192.168.202.128@tcp [ 7549.660617] Key type .llcrypt unregistered [ 7549.662614] Key type ._llcrypt unregistered [ 7603.346403] Key type ._llcrypt registered [ 7603.353065] Key type .llcrypt registered [ 7603.540058] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 7625.392533] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7627.506831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7627.593276] alg: No test for adler32 (adler32-zlib) [ 7628.932366] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 7629.475905] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 7631.248833] Key type lgssc registered [ 7632.813899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7690.361579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7690.407970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7691.707987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7691.753232] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7691.867270] Lustre: lustre-MDT0000: new disk, initializing [ 7692.025678] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7692.079276] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7696.889213] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7708.285846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7708.416585] Lustre: 296813:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7708.481239] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7708.487128] Lustre: Skipped 1 previous similar message [ 7708.573934] Lustre: lustre-MDT0001: new disk, initializing [ 7708.698272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7708.754043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7708.763749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7713.925669] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7723.919462] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7729.294745] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7736.184427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7736.656956] Lustre: lustre-OST2710: new disk, initializing [ 7736.676421] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7736.691622] Lustre: 298782:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7736.872229] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7743.112458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7743.122607] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7743.323146] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7743.934584] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7758.195750] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 7766.269574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7766.436446] Lustre: lustre-OST03e8: new disk, initializing [ 7766.439855] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7766.444879] Lustre: 299848:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7766.524215] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7773.636532] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7773.898413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7773.907567] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7774.068079] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7782.367810] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 7790.384643] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7790.742694] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7795.736821] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7795.925356] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7801.827907] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7802.098283] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7807.117443] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7807.333845] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7813.166368] Lustre: server umount lustre-MDT0000 complete [ 7815.139719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7815.161861] Lustre: Skipped 1 previous similar message [ 7816.802100] LustreError: 296803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492720 with bad export cookie 4752026252309089475 [ 7816.802974] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7816.818847] LustreError: 296803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7817.133143] Lustre: server umount lustre-MDT0001 complete [ 7831.297565] Lustre: server umount lustre-OST2710 complete [ 7844.603396] Lustre: server umount lustre-OST03e8 complete [ 7863.841586] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 7875.635847] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 7915.302340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7915.510343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7915.538536] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7915.616867] Lustre: lustre-MDT0000: new disk, initializing [ 7915.726774] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7915.773954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7920.189680] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7930.169494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7930.279716] Lustre: 305105:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7930.328463] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7930.332857] Lustre: Skipped 1 previous similar message [ 7930.411716] Lustre: lustre-MDT0001: new disk, initializing [ 7930.505866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7930.547367] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7930.566639] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7935.084670] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7946.272829] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7952.892368] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7960.903132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7961.303533] Lustre: lustre-OST0000: new disk, initializing [ 7961.307800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7961.313917] Lustre: 307075:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7961.418795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7968.125574] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 7971.427957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7971.447653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7971.592703] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7977.349359] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7982.045581] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7982.311128] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7986.807575] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7987.016215] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7992.290988] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7992.296263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7992.306631] LustreError: Skipped 1 previous similar message [ 7992.309884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7992.356376] Lustre: Skipped 3 previous similar messages [ 7997.426250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7997.431681] Lustre: Skipped 2 previous similar messages [ 8002.539025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8002.548088] Lustre: Skipped 1 previous similar message [ 8003.040869] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8003.306571] Lustre: server umount lustre-OST0000 complete [ 8012.776368] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8012.782153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8012.811257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8012.828348] Lustre: Skipped 1 previous similar message [ 8015.044299] Lustre: server umount lustre-MDT0000 complete [ 8017.889211] LustreError: 305116:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.910232] LustreError: 305116:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8019.927058] LustreError: 305096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782492923 with bad export cookie 4752026252309091155 [ 8019.936369] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8019.946217] LustreError: 305096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8020.520812] Lustre: server umount lustre-MDT0001 complete [ 8031.847233] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:55:33 (1782492933) [ 8033.672480] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8035.720105] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:55:38 (1782492938) [ 8039.332504] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8070.026712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8070.346975] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8070.368519] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8070.542487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8075.113459] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8083.823979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8083.943563] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8083.974800] Lustre: 310935:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8089.793656] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8111.794496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8112.132391] Lustre: Denying initial registration attempt from nid 192.168.202.128@tcp, specified as failover [ 8112.143873] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8112.161653] LustreError: 312387:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 8112.175216] LustreError: 312387:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8112.185267] LustreError: 312387:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8112.251611] Lustre: server umount lustre-OST0000 complete [ 8112.257587] LustreError: 312387:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8120.290150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8120.291928] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8120.312031] Lustre: Skipped 1 previous similar message [ 8120.318237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8120.324079] Lustre: Skipped 1 previous similar message [ 8126.149212] Lustre: server umount lustre-MDT0000 complete [ 8130.538280] LustreError: 312390:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8130.570319] LustreError: 312390:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8131.674129] LustreError: 310925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493035 with bad export cookie 4752026252309092037 [ 8131.680188] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8131.697518] LustreError: 310925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8132.102212] Lustre: server umount lustre-MDT0001 complete [ 8147.194549] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8150.225761] Key type lgssc unregistered [ 8150.601895] LNet: 313512:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8150.614289] LNetError: 313512:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8150.637874] LNet: Removed LNI 192.168.202.128@tcp [ 8151.751707] Key type .llcrypt unregistered [ 8151.757812] Key type ._llcrypt unregistered [ 8166.944355] Key type ._llcrypt registered [ 8166.946565] Key type .llcrypt registered [ 8167.042732] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8168.383933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8168.403491] alg: No test for adler32 (adler32-zlib) [ 8169.614906] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 8169.999939] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8171.696180] Key type lgssc registered [ 8173.509263] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8223.311422] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8232.036197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8232.087753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8233.368989] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8233.393593] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8233.552511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.551380] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8247.436515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8247.661199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8247.728924] Lustre: 316281:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8248.164191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8252.542305] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8259.849585] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8266.369789] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8288.903229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8289.308213] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8289.690143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8296.696695] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8306.165817] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8309.730264] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8309.730350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8309.738456] LustreError: Skipped 1 previous similar message [ 8309.741571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8309.756574] Lustre: Skipped 1 previous similar message [ 8309.771089] Lustre: Skipped 1 previous similar message [ 8314.856142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8314.869565] Lustre: Skipped 1 previous similar message [ 8315.226577] Lustre: server umount lustre-OST0000 complete [ 8325.094749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8325.097411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8325.102251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8325.139081] Lustre: Skipped 1 previous similar message [ 8326.893209] Lustre: server umount lustre-MDT0000 complete [ 8330.219153] LustreError: 316292:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8330.255343] LustreError: 316292:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8331.713682] LustreError: 317022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493235 with bad export cookie 14844280980327093829 [ 8331.727781] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8331.738581] LustreError: 317022:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8332.484614] Lustre: server umount lustre-MDT0001 complete [ 8346.765882] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8349.885832] Key type lgssc unregistered [ 8350.186907] LNet: 319854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8350.195102] LNetError: 319854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8350.232135] LNet: Removed LNI 192.168.202.128@tcp [ 8351.265359] Key type .llcrypt unregistered [ 8351.268800] Key type ._llcrypt unregistered [ 8383.967507] Key type ._llcrypt registered [ 8383.969310] Key type .llcrypt registered [ 8384.112453] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8386.060934] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8386.108984] alg: No test for adler32 (adler32-zlib) [ 8387.403755] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 8387.736606] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8389.448356] Key type lgssc registered [ 8390.774711] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8398.830455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8398.858511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8400.702757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8404.987584] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8411.107143] LustreError: 321285:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8411.129044] LustreError: 321285:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8414.074271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8414.592280] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8419.637197] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8427.368625] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8434.059361] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8444.599281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8445.105592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8453.441871] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8461.731975] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8469.868201] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 8469.992231] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8470.010794] Lustre: Skipped 2 previous similar messages [ 8474.975905] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [ 8474.993951] Lustre: Skipped 3 previous similar messages [ 8475.808364] Lustre: server umount lustre-MDT0000 complete [ 8479.725189] LustreError: 321267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493383 with bad export cookie 6399185425229498908 [ 8479.739562] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8480.100755] LustreError: 323779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8480.138811] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [ 8480.148837] Lustre: Skipped 3 previous similar messages [ 8480.225621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8484.257867] LustreError: 323779:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8484.258666] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8484.275162] LustreError: 323779:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8484.293602] Lustre: Skipped 1 previous similar message [ 8489.379630] LustreError: 321281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8489.379720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8489.392272] LustreError: 321281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8489.418122] Lustre: Skipped 1 previous similar message [ 8494.048241] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8494.512156] Lustre: server umount lustre-MDT0001 complete [ 8501.263940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8509.904964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8524.257118] LustreError: 320595:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000b0d7676d x1869079685148160/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8524.679753] LustreError: 323119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8524.704691] LustreError: 323119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8524.818371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8524.886354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8530.135911] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8536.037363] LustreError: 323120:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8536.071409] LustreError: 323120:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8540.466140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8540.946306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8545.036894] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8545.855656] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8553.117328] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8555.091295] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8560.871239] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8562.706390] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8570.852957] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8570.854460] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8570.875910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8570.897278] Lustre: Skipped 2 previous similar messages [ 8573.668244] Lustre: server umount lustre-OST0000 complete [ 8581.112206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.125119] Lustre: Skipped 2 previous similar messages [ 8590.310865] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8590.320373] Lustre: Skipped 5 previous similar messages [ 8593.376144] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8593.559183] Lustre: server umount lustre-MDT0000 complete [ 8595.425513] LustreError: 324884:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8595.441575] LustreError: 324884:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8597.089690] LustreError: 321266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493500 with bad export cookie 6399185425229506678 [ 8597.106777] LustreError: 321266:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8597.113977] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8597.527169] Lustre: server umount lustre-MDT0001 complete [ 8608.855574] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8611.747632] Key type lgssc unregistered [ 8612.009776] LNet: 327559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8612.015929] LNetError: 327559:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8613.031444] LNet: Removed LNI 192.168.202.128@tcp [ 8614.195398] Key type .llcrypt unregistered [ 8614.197476] Key type ._llcrypt unregistered [ 8638.892748] Key type ._llcrypt registered [ 8638.895979] Key type .llcrypt registered [ 8638.995360] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8640.172370] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8640.194677] alg: No test for adler32 (adler32-zlib) [ 8641.244831] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 8641.414591] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8643.080385] Key type lgssc registered [ 8644.389447] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8652.370789] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8652.384292] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8653.809171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8658.503442] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8664.042744] LustreError: 328976:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8664.057320] LustreError: 328976:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8669.163415] LustreError: 328977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8669.604973] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8670.270959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8677.194582] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8686.418341] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8691.738614] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8695.781501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8695.791650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8695.803772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8699.874791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8699.879268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8701.434804] Lustre: server umount lustre-MDT0000 complete [ 8705.003264] LustreError: 330216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8705.028752] LustreError: 330216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8705.622630] LustreError: 328956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493609 with bad export cookie 11423674268570644130 [ 8705.629148] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8705.633875] LustreError: 328956:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8705.936303] Lustre: server umount lustre-MDT0001 complete [ 8719.244677] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.553629] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8719.576922] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8719.726175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8723.722488] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8732.459352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.558153] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8732.595765] Lustre: 331566:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8733.025376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8737.679524] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8746.486342] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8751.676651] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8760.943461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8761.264229] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8761.275726] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8761.286734] LustreError: 333451:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 8761.293294] LustreError: 333451:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8761.296712] LustreError: 333451:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8761.411462] Lustre: server umount lustre-OST0000 complete [ 8761.413769] LustreError: 333451:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8773.040845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8773.263612] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8773.607289] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8781.221867] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8781.581777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8789.508972] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8798.361120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8798.547166] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8798.699440] Lustre: lustre-OST0001: new disk, initializing [ 8798.707629] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8798.731879] Lustre: 334991:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8798.858447] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8804.562251] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8804.570986] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8804.893535] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8805.077696] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 8814.621102] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8820.202679] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8820.204361] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8820.204903] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8820.204910] Lustre: Skipped 1 previous similar message [ 8820.227358] Lustre: Skipped 1 previous similar message [ 8820.251197] LustreError: Skipped 1 previous similar message [ 8823.983393] Lustre: server umount lustre-OST0001 complete [ 8825.324204] LustreError: 333917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8827.874444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8827.874833] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8827.889118] LustreError: Skipped 1 previous similar message [ 8827.891087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8827.904186] Lustre: Skipped 1 previous similar message [ 8827.913419] Lustre: Skipped 1 previous similar message [ 8830.440822] LustreError: 333919:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8830.466688] LustreError: 333919:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8835.564076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8835.578512] Lustre: Skipped 4 previous similar messages [ 8840.677060] LustreError: 333917:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8840.706751] LustreError: 333917:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8842.208570] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8842.503558] Lustre: server umount lustre-OST0000 complete [ 8850.914210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8850.915506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8850.930401] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8850.930410] Lustre: Skipped 2 previous similar messages [ 8850.958516] Lustre: Skipped 1 previous similar message [ 8853.524923] Lustre: server umount lustre-MDT0000 complete [ 8857.910897] LustreError: 331556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782493761 with bad export cookie 11423674268570644683 [ 8857.913847] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8857.924811] LustreError: 331556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8858.343831] Lustre: server umount lustre-MDT0001 complete [ 8868.581871] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 8871.306746] Key type lgssc unregistered [ 8871.557812] LNet: 336952:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8871.564860] LNetError: 336952:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8872.617324] LNet: Removed LNI 192.168.202.128@tcp [ 8873.668996] Key type .llcrypt unregistered [ 8873.684285] Key type ._llcrypt unregistered [ 8947.219769] Key type ._llcrypt registered [ 8947.223276] Key type .llcrypt registered [ 8947.334262] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 8963.218801] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 8964.933439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8965.106144] alg: No test for adler32 (adler32-zlib) [ 8966.511477] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 8967.011588] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 8968.716114] Key type lgssc registered [ 8970.460609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9010.130933] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9010.187186] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9011.503544] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9011.575311] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9011.685422] Lustre: lustre-MDT0000: new disk, initializing [ 9011.855632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9011.875686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9016.419770] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9027.874620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9028.071916] Lustre: 341892:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9028.137462] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9028.141912] Lustre: Skipped 1 previous similar message [ 9028.259686] Lustre: lustre-MDT0001: new disk, initializing [ 9028.431928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9028.480036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9028.493089] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9034.148533] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9045.102760] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9050.685655] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9058.773853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9059.210412] Lustre: lustre-OST0000: new disk, initializing [ 9059.216277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9059.223693] Lustre: 343860:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9059.335266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9065.431755] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9066.627761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9066.640037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9066.715689] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9074.721775] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9079.556069] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9079.786884] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9083.923785] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9084.067752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9087.463523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9087.464019] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9087.481023] Lustre: Skipped 1 previous similar message [ 9087.481282] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9092.581478] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9092.593040] Lustre: Skipped 2 previous similar messages [ 9097.719211] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9097.726406] Lustre: Skipped 1 previous similar message [ 9100.258612] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9100.403618] Lustre: server umount lustre-OST0000 complete [ 9105.378082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9105.390715] LustreError: Skipped 1 previous similar message [ 9105.394198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9105.415167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9107.943276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9110.780627] Lustre: server umount lustre-MDT0000 complete [ 9114.099141] LustreError: 343859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494017 with bad export cookie 1833166381962330127 [ 9114.100390] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9114.117849] LustreError: 343859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9114.441671] Lustre: server umount lustre-MDT0001 complete [ 9122.502996] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:13:45 (1782494025) [ 9129.209969] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:13:52 (1782494032) [ 9137.416310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9137.812233] LustreError: 346616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9137.834336] LustreError: 346616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9137.880242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.154599] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9143.276179] LustreError: 346617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9147.367983] LustreError: 346616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9150.376207] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9150.846395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9156.015592] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9164.573503] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9170.869960] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9177.577490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9178.031161] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9183.408086] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9190.690397] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9202.551427] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9204.409125] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9206.273647] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9208.314705] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9210.543071] Lustre: Failing over lustre-MDT0000 [ 9210.839070] Lustre: server umount lustre-MDT0000 complete [ 9210.856219] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9210.873595] LustreError: 346617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9210.889077] LustreError: 346617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9215.973162] LustreError: 347602:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9215.992957] LustreError: 347602:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9224.034714] LustreError: 346612:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9224.043523] LustreError: 346612:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9230.427331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9230.504263] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9230.710438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9230.755324] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9235.074751] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9235.293356] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9235.948708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9235.990391] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9236.053530] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9243.861256] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9245.401086] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9248.973592] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9251.198518] Lustre: Failing over lustre-MDT0000 [ 9251.301559] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9251.303561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9251.314200] Lustre: Skipped 3 previous similar messages [ 9251.318530] Lustre: Skipped 2 previous similar messages [ 9251.328198] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9251.502474] Lustre: server umount lustre-MDT0000 complete [ 9255.797976] LustreError: 347858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9255.809843] LustreError: 347858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9269.673307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9269.792526] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9270.049271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9270.108042] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9271.144157] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9274.126676] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9275.381993] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9275.388675] Lustre: Skipped 2 previous similar messages [ 9275.422979] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9275.445360] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9284.372374] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9285.913917] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9295.846661] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9295.849863] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9295.852654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9295.858135] Lustre: Skipped 3 previous similar messages [ 9295.875748] Lustre: Skipped 4 previous similar messages [ 9298.156178] Lustre: server umount lustre-MDT0000 complete [ 9300.966296] LustreError: 346617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9300.990285] LustreError: 346617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 13 previous similar messages [ 9301.656880] LustreError: 346596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494205 with bad export cookie 1833166381962332836 [ 9301.667529] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9301.669102] LustreError: 346596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9301.948565] Lustre: server umount lustre-MDT0001 complete [ 9316.348481] Lustre: server umount lustre-OST0000 complete [ 9334.234584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9334.658367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9338.989506] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9346.993666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9351.320514] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9358.092469] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9363.371454] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9369.454898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9375.422961] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9380.029124] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9384.255452] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9390.561653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9390.587845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9390.595462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9390.631635] Lustre: Skipped 1 previous similar message [ 9390.658913] Lustre: Skipped 1 previous similar message [ 9395.710194] LustreError: 354806:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9395.737319] LustreError: 354806:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9395.809447] Lustre: server umount lustre-OST0000 complete [ 9406.813425] Lustre: server umount lustre-MDT0000 complete [ 9411.547612] LustreError: 352964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494315 with bad export cookie 1833166381962333585 [ 9411.568661] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9411.575207] LustreError: 352964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9411.841838] Lustre: server umount lustre-MDT0001 complete [ 9422.479600] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 9425.334821] Key type lgssc unregistered [ 9425.631058] LNet: 356514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9425.647584] LNetError: 356514:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9425.663834] LNet: Removed LNI 192.168.202.128@tcp [ 9426.439579] Key type .llcrypt unregistered [ 9426.445705] Key type ._llcrypt unregistered [ 9449.946934] Key type ._llcrypt registered [ 9449.952222] Key type .llcrypt registered [ 9450.073783] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 9451.398925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9451.417971] alg: No test for adler32 (adler32-zlib) [ 9452.389330] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 9452.570366] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 9454.200405] Key type lgssc registered [ 9455.172378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9461.438898] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9461.456318] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9462.781374] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9466.444753] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9472.995245] LustreError: 358023:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9473.006215] LustreError: 358023:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9474.530603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9474.953255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9480.070558] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9486.642730] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9492.013587] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9499.173876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9499.698879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9506.586358] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9507.835202] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9513.122602] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9523.188049] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9523.194451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9523.202037] Lustre: Skipped 2 previous similar messages [ 9523.222382] Lustre: Skipped 2 previous similar messages [ 9524.706438] Lustre: server umount lustre-MDT0000 complete [ 9528.297614] LustreError: 358024:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9528.317960] LustreError: 358024:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9528.452702] LustreError: 358004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494432 with bad export cookie 11967477613293741911 [ 9528.455937] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9528.462948] LustreError: 358004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9528.702681] Lustre: server umount lustre-MDT0001 complete [ 9542.991178] Lustre: server umount lustre-OST0000 complete [ 9554.742781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9555.154241] LustreError: 361736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9555.262616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9559.006233] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9560.548466] LustreError: 361737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9564.644067] LustreError: 361736:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9568.361176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9568.872799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9572.936992] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9580.953685] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9586.246383] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9593.171836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9593.480932] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9599.679095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9600.065023] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9606.973639] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9619.939669] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9619.944912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9619.958914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9624.243783] Lustre: server umount lustre-MDT0000 complete [ 9625.578849] LustreError: 362463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9625.601347] LustreError: 362463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9627.978498] LustreError: 361717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494531 with bad export cookie 11967477613293743024 [ 9627.990533] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9627.994281] LustreError: 361717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9628.514413] Lustre: server umount lustre-MDT0001 complete [ 9643.843736] Lustre: server umount lustre-OST0000 complete [ 9665.353960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.023768] LustreError: 365646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9666.122080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9670.899996] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9679.877262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.643504] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9692.208610] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9696.740910] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9703.153037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9703.563794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9703.569714] Lustre: Skipped 1 previous similar message [ 9709.402838] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9711.797782] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9715.512335] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9720.289110] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9720.298664] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9720.309707] Lustre: Skipped 2 previous similar messages [ 9720.314284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9720.325070] Lustre: Skipped 3 previous similar messages [ 9722.341431] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9722.345258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9722.345645] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9722.364490] Lustre: Skipped 1 previous similar message [ 9724.871440] Lustre: server umount lustre-OST0000 complete [ 9731.554243] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9731.563540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.581613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9742.304357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9742.442083] Lustre: server umount lustre-MDT0000 complete [ 9742.818144] LustreError: 365646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9742.837706] LustreError: 365646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9745.229783] LustreError: 365627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494648 with bad export cookie 11967477613293744046 [ 9745.239523] LustreError: 365627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9745.247962] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9745.554710] Lustre: server umount lustre-MDT0001 complete [ 9754.812938] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 9757.543126] Key type lgssc unregistered [ 9757.862735] LNet: 369180:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9757.876361] LNetError: 369180:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9757.897850] LNet: Removed LNI 192.168.202.128@tcp [ 9758.977044] Key type .llcrypt unregistered [ 9758.979693] Key type ._llcrypt unregistered [ 9789.986670] Key type ._llcrypt registered [ 9789.990812] Key type .llcrypt registered [ 9790.128923] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 9791.638479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9791.917764] alg: No test for adler32 (adler32-zlib) [ 9793.652583] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 9794.124494] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 9795.856198] Key type lgssc registered [ 9796.876691] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9804.229469] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9804.247716] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9804.253339] LustreError: 370622:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9804.266406] LustreError: 370622:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9804.272066] LustreError: 370622:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9804.276516] LustreError: 370622:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9804.281766] LustreError: 370622:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9811.544500] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9811.550637] LustreError: 371020:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9811.570616] LustreError: 371020:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9811.580713] LustreError: 371020:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9811.587779] LustreError: 371020:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9811.593691] LustreError: 371020:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9829.627800] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [ 9832.345536] Key type lgssc unregistered [ 9832.663559] LNet: 372023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9832.679303] LNetError: 372023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9832.699171] LNet: Removed LNI 192.168.202.128@tcp [ 9833.526172] Key type .llcrypt unregistered [ 9833.529213] Key type ._llcrypt unregistered [ 9851.869501] Key type ._llcrypt registered [ 9851.872364] Key type .llcrypt registered [ 9851.985418] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [ 9864.456754] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [ 9865.631989] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9865.718736] alg: No test for adler32 (adler32-zlib) [ 9866.733147] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [ 9866.961672] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [ 9868.640450] Key type lgssc registered [ 9869.579750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9902.953156] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9902.991276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9904.237312] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9904.279892] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9904.381633] Lustre: lustre-MDT0000: new disk, initializing [ 9904.482388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9904.497457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9908.790900] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9918.356628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9918.496550] Lustre: 375164:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9918.581484] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9918.600382] Lustre: Skipped 1 previous similar message [ 9918.717390] Lustre: lustre-MDT0001: new disk, initializing [ 9918.825181] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9918.886992] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9918.892429] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9923.792633] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9932.511632] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9937.313459] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9943.661679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9943.957023] Lustre: lustre-OST0000: new disk, initializing [ 9943.961331] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9943.971622] Lustre: 377130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9944.096188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9950.170483] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [ 9953.322250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9953.336509] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9953.472206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9958.347887] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9962.516482] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9962.680412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9967.216576] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9967.539405] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9972.201381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9972.230171] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9972.238292] Lustre: Skipped 1 previous similar message [ 9975.976441] Lustre: server umount lustre-OST0000 complete [ 9980.385235] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9980.390438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9980.399963] Lustre: Skipped 1 previous similar message [ 9980.407143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.435030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9982.435747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9985.548782] Lustre: server umount lustre-MDT0000 complete [ 9988.469370] LustreError: 375154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782494892 with bad export cookie 9361321428700646573 [ 9988.481371] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9988.482357] LustreError: 375154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9988.749611] Lustre: server umount lustre-MDT0001 complete [ 9995.537625] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9996.908176] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:28:20 (1782494900) [10002.380449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10002.697306] LustreError: 379589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10002.720201] LustreError: 379589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10002.780278] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10006.375124] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10008.046039] LustreError: 379590:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.155321] LustreError: 379589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.909729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.252855] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10018.325369] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10025.037720] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10029.670519] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10034.950259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10035.312618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10042.164794] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10048.956460] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10055.807248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10055.949036] Lustre: lustre-OST0001: new disk, initializing [10055.953671] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10055.959156] Lustre: 382422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10056.040136] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10062.140854] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10064.503653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10064.512634] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10064.628185] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10070.331194] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10075.109320] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10075.120609] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10075.141451] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10075.147254] LustreError: Skipped 1 previous similar message [10075.154449] Lustre: Skipped 2 previous similar messages [10080.176932] Lustre: server umount lustre-OST0001 complete [10080.231085] LustreError: 381410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10080.238913] LustreError: 381410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10085.347190] LustreError: 381410:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10085.362433] LustreError: 381410:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10095.594573] LustreError: 381409:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10095.618220] LustreError: 381409:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10099.174867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10099.185358] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10099.205232] Lustre: Skipped 1 previous similar message [10099.212985] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10099.220158] Lustre: Skipped 1 previous similar message [10103.332384] Lustre: server umount lustre-OST0000 complete [10110.947044] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.967025] Lustre: Skipped 1 previous similar message [10110.972742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10110.987776] Lustre: Skipped 3 previous similar messages [10121.184965] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10121.802934] Lustre: server umount lustre-MDT0000 complete [10126.315637] LustreError: 379589:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10126.345823] LustreError: 379589:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10126.432782] LustreError: 379571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495030 with bad export cookie 9361321428700647455 [10126.435340] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10126.449357] LustreError: 379571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10133.088686] Lustre: server umount lustre-MDT0001 complete [10143.009703] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10145.642683] Key type lgssc unregistered [10145.893190] LNet: 384413:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10145.904327] LNetError: 384413:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10145.918834] LNet: Removed LNI 192.168.202.128@tcp [10146.931880] Key type .llcrypt unregistered [10146.934717] Key type ._llcrypt unregistered [10195.372755] Key type ._llcrypt registered [10195.375532] Key type .llcrypt registered [10195.496849] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [10196.717730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10196.736569] alg: No test for adler32 (adler32-zlib) [10197.822081] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [10198.030588] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [10199.688387] Key type lgssc registered [10200.925447] Lustre: Echo OBD driver; http://www.lustre.org/ [10208.637830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10208.660116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.034130] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10210.074341] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10210.257455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10215.174604] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10224.283112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10224.352611] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10224.378156] Lustre: 387211:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10224.622593] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10228.473908] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10234.482223] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10240.348255] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10245.089516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10245.091027] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10245.096214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10245.126920] Lustre: Skipped 1 previous similar message [10248.387702] Lustre: server umount lustre-MDT0000 complete [10251.171499] LustreError: 387204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495154 with bad export cookie 3707492928010369194 [10251.175747] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10251.179322] LustreError: 387204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10251.515734] Lustre: server umount lustre-MDT0001 complete [10259.621548] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:32:42 (1782495162) [10265.411464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10265.664171] LustreError: 389722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10265.678134] LustreError: 389722:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10265.723563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10270.109577] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10271.213614] LustreError: 389723:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10276.334453] LustreError: 389722:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10278.063733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10278.387272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10282.602417] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10289.256622] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10294.424741] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10300.921648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10301.134431] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10301.422572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10309.183845] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10317.331067] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10325.893723] Lustre: Permanently deactivating lustre-OST0000 [10325.916924] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10331.717221] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10334.206795] LustreError: 392657:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10334.214887] LustreError: 392657:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [10334.219511] LustreError: 392657:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10337.727543] LustreError: 392706:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [10337.732119] LustreError: 392706:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10345.940064] Lustre: server umount lustre-OST0000 complete [10348.903553] LustreError: 392956:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [10348.909374] LustreError: 392956:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10353.646611] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10353.652073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10353.654426] Lustre: Skipped 1 previous similar message [10353.673536] Lustre: Skipped 2 previous similar messages [10358.753705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10358.758870] Lustre: Skipped 1 previous similar message [10360.881020] Lustre: server umount lustre-MDT0000 complete [10363.878481] LustreError: 389717:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10363.912534] LustreError: 389717:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10364.061242] LustreError: 392410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495267 with bad export cookie 3707492928010369775 [10364.074650] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10364.076074] LustreError: 392410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10374.025860] Lustre: server umount lustre-MDT0001 complete [10383.650281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10388.640427] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10392.299234] LustreError: 394114:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10400.541931] Lustre: 394456:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.128@tcp [10402.920952] Lustre: server umount MGS complete [10415.066709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.400968] LustreError: 395180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.419913] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10415.424161] Lustre: Skipped 1 previous similar message [10415.479839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10420.342591] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10425.837094] LustreError: 395181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10425.864585] LustreError: 395181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10429.771235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10430.142240] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10435.233956] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10442.281313] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10447.380794] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10448.866921] Lustre: Permanently reactivating lustre-OST0000 [10448.888649] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10448.891922] Lustre: Skipped 2 previous similar messages [10477.536216] LustreError: 396970:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10487.499744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10488.130213] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10488.134890] Lustre: Skipped 1 previous similar message [10497.586632] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10506.957463] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10512.868900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10512.899450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10512.950839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10513.894604] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10514.156734] Lustre: server umount lustre-OST0000 complete [10519.012529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10519.031164] Lustre: Skipped 1 previous similar message [10519.045196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10519.065975] Lustre: Skipped 2 previous similar messages [10529.250900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10529.258587] Lustre: Skipped 4 previous similar messages [10533.344806] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10533.753528] Lustre: server umount lustre-MDT0000 complete [10534.369159] LustreError: 395180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10534.389239] LustreError: 395180:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10538.106440] LustreError: 396754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495441 with bad export cookie 3707492928010370818 [10538.110851] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10538.121596] LustreError: 396754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10538.518704] Lustre: server umount lustre-MDT0001 complete [10550.046216] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10552.404034] Key type lgssc unregistered [10552.654946] LNet: 399595:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10552.671059] LNetError: 399595:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10552.688602] LNet: Removed LNI 192.168.202.128@tcp [10553.511743] Key type .llcrypt unregistered [10553.514685] Key type ._llcrypt unregistered [10572.860850] Key type ._llcrypt registered [10572.863775] Key type .llcrypt registered [10572.970833] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [10587.543655] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [10588.412657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10588.484890] alg: No test for adler32 (adler32-zlib) [10589.539716] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [10589.729988] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [10591.384351] Key type lgssc registered [10592.567071] Lustre: Echo OBD driver; http://www.lustre.org/ [10631.346751] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10631.387951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10632.803934] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10632.951841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10633.082923] Lustre: lustre-MDT0000: new disk, initializing [10633.259897] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10633.299938] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10637.675985] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10648.963258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.104468] Lustre: 402753:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10649.181420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10649.188886] Lustre: Skipped 1 previous similar message [10649.266411] Lustre: lustre-MDT0001: new disk, initializing [10649.366698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10649.409113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10649.422520] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10654.878672] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10665.183408] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10672.313244] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10680.449204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10680.692457] Lustre: lustre-OST0000: new disk, initializing [10680.695789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10680.704911] Lustre: 404720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10680.783564] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10687.141486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10687.153451] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10687.270520] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10688.751307] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10700.566393] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10706.103787] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10706.346184] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10711.529684] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10711.723457] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10718.187555] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10718.211264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.233911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10718.244721] Lustre: Skipped 1 previous similar message [10720.126931] Lustre: server umount lustre-OST0000 complete [10726.371071] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10726.381842] LustreError: Skipped 1 previous similar message [10726.388470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10726.404872] Lustre: Skipped 1 previous similar message [10726.412695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10728.417525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10728.417701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10733.538814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10733.547757] Lustre: Skipped 2 previous similar messages [10738.144147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10738.355389] Lustre: server umount lustre-MDT0000 complete [10742.222965] LustreError: 403546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495645 with bad export cookie 7300412155258626133 [10742.224463] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10742.239554] LustreError: 403546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10742.727952] Lustre: server umount lustre-MDT0001 complete [10751.355953] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10753.239158] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:40:56 (1782495656) [10761.408470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10762.039321] LustreError: 407185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10762.060017] LustreError: 407185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10762.148538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10766.987146] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10767.343735] LustreError: 407186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10771.494507] LustreError: 407185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10776.544916] LustreError: 407186:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10776.781569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.319725] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10782.075858] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10788.604342] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10794.912322] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10801.404165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10801.749960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10808.427256] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10814.486501] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10816.903646] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10819.505163] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10827.747891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10827.755976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10827.762165] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10827.763279] Lustre: Skipped 1 previous similar message [10827.782887] Lustre: Skipped 1 previous similar message [10832.080489] Lustre: server umount lustre-OST0000 complete [10837.986457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10838.019605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10838.033745] Lustre: Skipped 1 previous similar message [10842.599219] Lustre: server umount lustre-MDT0000 complete [10843.105714] LustreError: 407185:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10843.138988] LustreError: 407185:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10846.396141] LustreError: 409957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495750 with bad export cookie 7300412155258627015 [10846.400579] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10846.413347] LustreError: 409957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10846.997733] Lustre: server umount lustre-MDT0001 complete [10862.083538] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10865.341340] Key type lgssc unregistered [10865.700630] LNet: 410912:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10865.712291] LNetError: 410912:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10865.731903] LNet: Removed LNI 192.168.202.128@tcp [10866.987258] Key type .llcrypt unregistered [10866.990761] Key type ._llcrypt unregistered [10893.716747] Key type ._llcrypt registered [10893.718650] Key type .llcrypt registered [10893.779401] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [10894.027796] Key type .llcrypt unregistered [10894.035677] Key type ._llcrypt unregistered [10910.888973] Key type ._llcrypt registered [10910.891737] Key type .llcrypt registered [10910.975491] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [10912.333691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10912.365352] alg: No test for adler32 (adler32-zlib) [10913.398434] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [10913.698068] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [10915.384242] Key type lgssc registered [10916.819973] Lustre: Echo OBD driver; http://www.lustre.org/ [10925.592459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10925.661690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10927.402386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10934.120269] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10937.828590] LustreError: 413126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10937.868036] LustreError: 413126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10942.956922] LustreError: 413127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10944.738133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.190754] LustreError: 413126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10945.242582] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10950.628370] LustreError: 413127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.660211] LustreError: 413127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10952.706032] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10952.716273] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10957.800187] LustreError: 413126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10957.840269] LustreError: 413126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10960.301611] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10967.217782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10967.587213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10971.483907] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [10982.595737] Lustre: server umount lustre-OST0000 complete [10987.069523] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10987.082513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10987.092819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.003474] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10988.008036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10988.066604] Lustre: Skipped 1 previous similar message [10993.124853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10993.128724] Lustre: Skipped 1 previous similar message [10997.318185] Lustre: server umount lustre-MDT0000 complete [10998.243924] LustreError: 413127:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10998.275224] LustreError: 413127:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10999.265375] Lustre: 412438:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782495886/real 1782495886] req@00000000b7d4d4aa x1869082333834240/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782495902 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10999.277025] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10999.301471] Lustre: 412438:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [11001.438913] LustreError: 413106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782495905 with bad export cookie 5485363647231936563 [11001.451120] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11001.454164] LustreError: 413106:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11001.857114] Lustre: server umount lustre-MDT0001 complete [11012.846667] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11015.633866] Key type lgssc unregistered [11015.996045] LNet: 416274:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11016.000841] LNetError: 416274:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11017.077259] LNet: Removed LNI 192.168.202.128@tcp [11018.091569] Key type .llcrypt unregistered [11018.095694] Key type ._llcrypt unregistered [11041.055744] Key type ._llcrypt registered [11041.057552] Key type .llcrypt registered [11041.159071] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11042.362823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11042.429073] alg: No test for adler32 (adler32-zlib) [11043.589033] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [11043.905787] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11045.568905] Key type lgssc registered [11046.643630] Lustre: Echo OBD driver; http://www.lustre.org/ [11054.732149] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11054.765608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11070.369784] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11075.488471] LustreError: 417668:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.128@tcp: failed processing log, type 4: rc = -110 [11102.240858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11108.762359] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11116.829286] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11124.569981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11124.947029] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11125.022402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11129.388863] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11130.339382] LustreError: 417694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.359954] LustreError: 417694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11135.470100] LustreError: 417693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11136.491582] LustreError: 417694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11136.510821] LustreError: 417694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11137.532202] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11137.835560] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11141.835923] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11148.356549] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11153.383037] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11161.058124] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11161.067574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11161.078600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11163.625019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11163.628744] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11163.631539] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11165.721548] Lustre: server umount lustre-OST0000 complete [11168.741418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11168.747502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11168.758475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11168.769466] Lustre: Skipped 2 previous similar messages [11173.859562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11173.866586] Lustre: Skipped 1 previous similar message [11175.351487] Lustre: server umount lustre-MDT0000 complete [11178.912489] LustreError: 417676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782496082 with bad export cookie 5704316950862669688 [11178.924549] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11179.235773] Lustre: server umount lustre-MDT0001 complete [11188.869189] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11191.566494] Key type lgssc unregistered [11191.879538] LNet: 421237:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11191.894292] LNetError: 421237:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11191.919080] LNet: Removed LNI 192.168.202.128@tcp [11192.893719] Key type .llcrypt unregistered [11192.895927] Key type ._llcrypt unregistered [11214.576477] Key type ._llcrypt registered [11214.578629] Key type .llcrypt registered [11214.691746] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11216.036673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11216.051155] alg: No test for adler32 (adler32-zlib) [11217.106799] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [11217.324810] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11218.984225] Key type lgssc registered [11220.142961] Lustre: Echo OBD driver; http://www.lustre.org/ [11227.183902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11227.197508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11228.596772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11232.687151] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11238.882474] LustreError: 422660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11238.904044] LustreError: 422660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11240.730369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11241.123055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11245.588375] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11252.824878] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11258.843848] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11265.440937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11265.960046] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11272.327984] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11274.196046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11281.192360] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11287.012199] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11287.016789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11287.029542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11288.969157] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.28@tcp (stopping) [11289.578951] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11289.592470] Lustre: Skipped 1 previous similar message [11291.877782] Lustre: server umount lustre-MDT0000 complete [11294.064959] LustreError: 422655:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11295.259957] LustreError: 422641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.28@tcp arrived at 1782496199 with bad export cookie 15971571034509347096 [11295.273663] LustreError: 422641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11297.069721] LustreError: 422639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782496200 with bad export cookie 15971571034509346200 [11297.092208] LustreError: 422639:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11297.096746] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11297.774933] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11297.781235] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11297.796242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11297.803349] Lustre: Skipped 3 previous similar messages [11299.818401] LustreError: 423388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11299.830198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11299.864512] LustreError: 423388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [11302.881890] LustreError: 422654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11302.902436] LustreError: 422654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11303.261172] Lustre: server umount lustre-OST0000 complete [11306.981745] LustreError: 423388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11306.997029] LustreError: 423388:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11309.495169] Lustre: server umount lustre-MDT0001 complete [11319.411794] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11322.095156] Key type lgssc unregistered [11322.257706] LNet: 426344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11322.272274] LNetError: 426344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11323.306859] LNet: Removed LNI 192.168.202.128@tcp [11324.121530] Key type .llcrypt unregistered [11324.123491] Key type ._llcrypt unregistered [11347.901493] Key type ._llcrypt registered [11347.904608] Key type .llcrypt registered [11347.986739] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11349.329507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11349.350236] alg: No test for adler32 (adler32-zlib) [11350.388452] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [11350.596689] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11352.368477] Key type lgssc registered [11353.991810] Lustre: Echo OBD driver; http://www.lustre.org/ [11363.123894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11363.166131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11364.923027] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11369.202660] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11375.089676] LustreError: 427766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11375.099807] LustreError: 427766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11378.150620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11378.747603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11383.857238] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11390.783776] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11396.027902] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11403.515476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11403.965737] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11409.913460] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11410.113063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11416.658554] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11424.630888] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [11424.738287] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11424.749468] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11425.763363] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11425.766720] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11425.787543] Lustre: Skipped 2 previous similar messages [11427.564867] Lustre: server umount lustre-MDT0001 complete [11429.742771] LustreError: 427761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.202.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.883313] LustreError: 427766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.899661] LustreError: 427766:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11436.001513] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11436.005849] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11436.016173] LustreError: 427765:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11436.041216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11440.836769] Lustre: server umount lustre-OST0000 complete [11441.125539] LustreError: 427766:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11444.993524] Lustre: server umount lustre-MDT0000 complete [11460.879897] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11463.594824] Key type lgssc unregistered [11463.852691] LNet: 431447:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11463.859215] LNetError: 431447:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11463.880146] LNet: Removed LNI 192.168.202.128@tcp [11464.534537] Key type .llcrypt unregistered [11464.536615] Key type ._llcrypt unregistered [11490.485328] Key type ._llcrypt registered [11490.487276] Key type .llcrypt registered [11490.641977] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [11504.534719] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11505.697266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11505.814615] alg: No test for adler32 (adler32-zlib) [11506.807674] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [11507.010334] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11508.673676] Key type lgssc registered [11509.884967] Lustre: Echo OBD driver; http://www.lustre.org/ [11542.574951] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11558.016364] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11558.060884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.419964] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11559.479175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11559.588303] Lustre: lustre-MDT0000: new disk, initializing [11559.686728] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11559.703258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11563.865655] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11576.067824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11576.268439] Lustre: lustre-OST0000: new disk, initializing [11576.271462] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11576.275219] Lustre: Skipped 1 previous similar message [11576.279870] Lustre: 436492:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11576.354110] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11576.462369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11576.479121] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11576.530314] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11582.809516] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11591.611498] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11605.438414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11605.568338] Lustre: 435476:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11605.634234] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11605.715821] Lustre: lustre-MDT0001: new disk, initializing [11605.835980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11605.886772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11605.902674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11610.074584] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11618.746793] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11618.915535] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11622.385409] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11622.550250] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11628.929428] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.28@tcp (stopping) [11630.050511] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11630.069973] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11630.071389] Lustre: Skipped 1 previous similar message [11630.073517] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11630.084922] Lustre: Skipped 1 previous similar message [11633.633815] Lustre: server umount lustre-MDT0001 complete [11640.297553] LustreError: 437011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11640.311127] LustreError: 437011:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11645.421681] LustreError: 435482:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11645.448902] LustreError: 435482:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11650.534496] LustreError: 438256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11650.554467] LustreError: 438256:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11653.601457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11653.611985] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11653.637037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11655.661101] LustreError: 435487:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11655.662450] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11660.772464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11660.774145] LustreError: 438256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11664.864552] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11665.117719] Lustre: server umount lustre-OST0000 complete [11683.296174] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11683.563436] Lustre: server umount lustre-MDT0000 complete [11696.352325] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11698.969693] Key type lgssc unregistered [11699.283653] LNet: 440240:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11699.287806] LNetError: 440240:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11699.303353] LNet: Removed LNI 192.168.202.128@tcp [11700.324859] Key type .llcrypt unregistered [11700.327674] Key type ._llcrypt unregistered [11743.057562] Key type ._llcrypt registered [11743.059948] Key type .llcrypt registered [11743.161777] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [11743.400450] Key type .llcrypt unregistered [11743.403568] Key type ._llcrypt unregistered [11762.171840] Key type ._llcrypt registered [11762.174041] Key type .llcrypt registered [11762.302761] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [11763.349923] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11763.809921] alg: No test for adler32 (adler32-zlib) [11764.838230] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [11765.137062] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [11766.912194] Key type lgssc registered [11768.197466] Lustre: Echo OBD driver; http://www.lustre.org/ [11795.423352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11795.478155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11796.810395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11796.850064] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11796.967673] Lustre: lustre-MDT0000: new disk, initializing [11797.119086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11797.172753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11801.956224] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11813.011436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11813.147762] Lustre: 443616:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11813.309584] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11813.312632] Lustre: Skipped 1 previous similar message [11813.418439] Lustre: lustre-MDT0001: new disk, initializing [11813.534794] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11813.573262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11813.587413] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11817.318831] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11825.953204] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11831.697735] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11838.700519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11838.887634] Lustre: lustre-OST0000: new disk, initializing [11838.894510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11838.909519] Lustre: 445587:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11839.000562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11844.504124] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11844.719751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11844.730110] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11844.869416] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11852.803636] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11859.939140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11859.951605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11859.991189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11865.576549] LustreError: 443625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11865.591897] LustreError: 443625:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11865.943193] Lustre: server umount lustre-MDT0000 complete [11869.993962] LustreError: 444413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782496773 with bad export cookie 6047865513091622220 [11870.000401] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11870.009622] LustreError: 444413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11870.442668] Lustre: server umount lustre-MDT0001 complete [11884.798579] Lustre: server umount lustre-OST0000 complete [11894.821254] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:59:58 (1782496798) [11902.522854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11902.829745] LustreError: 447614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.854955] LustreError: 447614:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11902.899359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11907.040397] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11908.072631] LustreError: 447615:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11913.193610] LustreError: 447614:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11914.810943] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11915.230179] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11919.737738] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11927.494538] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11933.129986] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11951.856430] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11952.019067] Lustre: Found index 0 for lustre-OST0000, updating log [11952.254899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11959.852446] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [11967.036764] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11973.744215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11984.352993] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11984.358536] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11984.368168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11984.368177] Lustre: Skipped 3 previous similar messages [11984.392038] Lustre: Skipped 3 previous similar messages [11985.443515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11985.458963] Lustre: Skipped 1 previous similar message [11990.361942] Lustre: server umount lustre-OST0000 complete [12006.883629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12006.908481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12006.911312] Lustre: Skipped 1 previous similar message [12012.001761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.010086] Lustre: Skipped 1 previous similar message [12015.072170] LNet: 1 local NIs in recovery (showing 1): 192.168.202.128@tcp [12020.192950] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12020.385111] Lustre: server umount lustre-MDT0000 complete [12024.199201] LustreError: 449469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782496927 with bad export cookie 6047865513091623613 [12024.203949] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12024.222081] LustreError: 449469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12024.665162] Lustre: server umount lustre-MDT0001 complete [12033.923813] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:02:17 (1782496937) [12653.852265] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12654.405389] LustreError: 454063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12654.485702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12659.287935] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12659.689103] LustreError: 454064:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12664.821622] LustreError: 454063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12667.627798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12668.004387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12672.500598] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12679.325820] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12684.778254] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12704.185457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12704.561773] Lustre: Found index 0 for lustre-OST0000, updating log [12705.004285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12706.402261] LNet: 1 local NIs in recovery (showing 1): 192.168.202.128@tcp [12712.293819] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [12720.585868] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12723.451488] Lustre: server umount lustre-OST0000 complete [12728.548446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12728.553420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12728.563726] Lustre: Skipped 2 previous similar messages [12728.597245] Lustre: Skipped 7 previous similar messages [12729.570459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12729.575490] Lustre: Skipped 1 previous similar message [12731.619592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12731.628803] Lustre: Skipped 3 previous similar messages [12734.049675] Lustre: server umount lustre-MDT0000 complete [12734.691509] LustreError: 454063:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12734.722401] LustreError: 454063:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12738.796634] LustreError: 454793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12738.836129] LustreError: 454793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [12738.976251] LustreError: 454043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782497642 with bad export cookie 6047865513091624698 [12738.985575] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12738.985710] LustreError: 454043:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12739.466573] Lustre: server umount lustre-MDT0001 complete [12766.816115] LNet: 1 local NIs in recovery (showing 1): 192.168.202.128@tcp [12771.074702] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [12780.834814] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [12816.067537] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:15:18 (1782497718) [12833.998452] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [12837.113945] Key type lgssc unregistered [12837.573447] LNet: 460671:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12837.579601] LNetError: 460671:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12965.609282] LNet: Removed LNI 192.168.202.128@tcp [12966.564131] Key type .llcrypt unregistered [12966.567091] Key type ._llcrypt unregistered [12985.176744] Key type ._llcrypt registered [12985.178268] Key type .llcrypt registered [12985.309528] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [12987.285718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12987.593000] alg: No test for adler32 (adler32-zlib) [12988.720392] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [12989.209327] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [12991.032223] Key type lgssc registered [12992.826599] Lustre: Echo OBD driver; http://www.lustre.org/ [12996.833969] LNet: Added LNI 192.168.202.128@tcp100 [8/256/0/180] [12997.585972] LNet: Added LNI 192.168.202.128@tcp101 [8/256/0/180] [12998.445112] LNet: Added LNI 192.168.202.128@tcp102 [8/256/0/180] [12999.255840] LNet: Added LNI 192.168.202.128@tcp103 [8/256/0/180] [13000.065801] LNet: Added LNI 192.168.202.128@tcp104 [8/256/0/180] [13000.842751] LNet: Added LNI 192.168.202.128@tcp105 [8/256/0/180] [13001.588466] LNet: Added LNI 192.168.202.128@tcp106 [8/256/0/180] [13002.366379] LNet: Added LNI 192.168.202.128@tcp107 [8/256/0/180] [13003.124065] LNet: Added LNI 192.168.202.128@tcp108 [8/256/0/180] [13003.902177] LNet: Added LNI 192.168.202.128@tcp109 [8/256/0/180] [13004.608907] LNet: Added LNI 192.168.202.128@tcp110 [8/256/0/180] [13005.338428] LNet: Added LNI 192.168.202.128@tcp111 [8/256/0/180] [13006.009595] LNet: Added LNI 192.168.202.128@tcp112 [8/256/0/180] [13006.794898] LNet: Added LNI 192.168.202.128@tcp113 [8/256/0/180] [13007.672342] LNet: Added LNI 192.168.202.128@tcp114 [8/256/0/180] [13008.374391] LNet: Added LNI 192.168.202.128@tcp115 [8/256/0/180] [13009.099523] LNet: Added LNI 192.168.202.128@tcp116 [8/256/0/180] [13009.898243] LNet: Added LNI 192.168.202.128@tcp117 [8/256/0/180] [13010.913991] LNet: Added LNI 192.168.202.128@tcp118 [8/256/0/180] [13011.965164] LNet: Added LNI 192.168.202.128@tcp119 [8/256/0/180] [13013.126751] LNet: Added LNI 192.168.202.128@tcp120 [8/256/0/180] [13014.037171] LNet: Added LNI 192.168.202.128@tcp121 [8/256/0/180] [13015.002788] LNet: Added LNI 192.168.202.128@tcp122 [8/256/0/180] [13016.034940] LNet: Added LNI 192.168.202.128@tcp123 [8/256/0/180] [13016.917977] LNet: Added LNI 192.168.202.128@tcp124 [8/256/0/180] [13017.814298] LNet: Added LNI 192.168.202.128@tcp125 [8/256/0/180] [13018.712810] LNet: Added LNI 192.168.202.128@tcp126 [8/256/0/180] [13019.595874] LNet: Added LNI 192.168.202.128@tcp127 [8/256/0/180] [13020.401974] LNet: Added LNI 192.168.202.128@tcp128 [8/256/0/180] [13021.263553] LNet: Added LNI 192.168.202.128@tcp129 [8/256/0/180] [13022.063290] LNet: Added LNI 192.168.202.128@tcp130 [8/256/0/180] [13022.987585] LNet: Added LNI 192.168.202.128@tcp131 [8/256/0/180] [13023.959253] LNet: Added LNI 192.168.202.128@tcp132 [8/256/0/180] [13024.762925] LNet: Added LNI 192.168.202.128@tcp133 [8/256/0/180] [13025.775960] LNet: Added LNI 192.168.202.128@tcp134 [8/256/0/180] [13026.690627] LNet: Added LNI 192.168.202.128@tcp135 [8/256/0/180] [13033.591171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13033.650766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13034.792619] Lustre: 463707:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [13034.956289] LNetError: 461169:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [13034.979807] LNetError: Protocol error connecting to 192.168.202.128@tcp121 at host 192.168.202.128:988: is it running a compatible version of Lustre? [13034.991886] LNetError: Skipped 5 previous similar messages [13035.099050] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13035.161734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13035.279451] Lustre: lustre-MDT0000: new disk, initializing [13035.302035] LNet: 6 local NIs in recovery (showing 6): 192.168.202.128@tcp126, 192.168.202.128@tcp127, 192.168.202.128@tcp128, 192.168.202.128@tcp125, 192.168.202.128@tcp121, 192.168.202.128@tcp129 [13035.437739] Lustre: 463763:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [13035.449927] Lustre: 463763:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13035.486669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13035.547397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13036.066069] LNetError: 463739:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [13036.075863] LNetError: 463739:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 18 previous similar messages [13036.086167] LNetError: Protocol error connecting to 192.168.202.128@tcp120 at host 192.168.202.128:988: is it running a compatible version of Lustre? [13036.103528] LNetError: Skipped 13 previous similar messages [13037.424560] LNetError: 461170:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [13037.461029] LNetError: Protocol error connecting to 192.168.202.128@tcp126 at host 192.168.202.128:988: is it running a compatible version of Lustre? [13037.492422] LNetError: Skipped 4 previous similar messages [13040.790357] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13041.570051] LNetError: 463738:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [13041.593108] LNetError: 463738:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [13041.603539] LNetError: Protocol error connecting to 192.168.202.128@tcp126 at host 192.168.202.128:988: is it running a compatible version of Lustre? [13049.764544] LNetError: 461167:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [13049.772328] LNetError: Protocol error connecting to 192.168.202.128@tcp126 at host 192.168.202.128:988: is it running a compatible version of Lustre? [13051.841904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13052.014931] Lustre: 463724:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13052.108099] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13052.114742] Lustre: Skipped 1 previous similar message [13052.302763] Lustre: lustre-MDT0001: new disk, initializing [13052.375995] Lustre: 464526:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [13052.385065] Lustre: 464526:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13052.536295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13052.604372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13052.619173] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13056.521387] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13059.122802] Lustre: 464905:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [13059.133076] Lustre: 464905:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13064.125319] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13065.123766] LNetError: 463732:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [13065.134176] LNetError: Protocol error connecting to 192.168.202.128@tcp126 at host 192.168.202.128:988: is it running a compatible version of Lustre? [13069.100992] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13075.091240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13075.389710] Lustre: lustre-OST0000: new disk, initializing [13075.392741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13075.400315] Lustre: 465697:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13075.485506] Lustre: 465702:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [13075.491670] Lustre: 465702:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [13075.524571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13080.585417] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13081.389542] LNetError: 461170:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [13081.399693] LNetError: 461170:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 16 previous similar messages [13081.414454] LNetError: Protocol error connecting to 192.168.202.128@tcp128 at host 192.168.202.128:988: is it running a compatible version of Lustre? [13081.428560] LNetError: Skipped 16 previous similar messages [13084.205648] Lustre: 466210:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.128@tcp: can't add NID 192.168.202.128@tcp130: rc = -75 [13084.222223] Lustre: 466210:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13084.311049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13084.323877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13084.430149] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13088.956721] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13095.856143] LNet: 1 local NIs in recovery (showing 1): 192.168.202.128@tcp126 [13100.001511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13100.004692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13100.004760] LustreError: Skipped 1 previous similar message [13100.013230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13100.026514] Lustre: Skipped 1 previous similar message [13102.610350] Lustre: server umount lustre-OST0000 complete [13108.719877] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13108.723901] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13108.730532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13108.735549] Lustre: Skipped 2 previous similar messages [13110.241529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13110.246558] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13110.269843] Lustre: Skipped 1 previous similar message [13112.832873] Lustre: server umount lustre-MDT0000 complete [13116.195228] LNetError: 461167:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.202.128 [13116.204641] LNetError: 461167:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 29 previous similar messages [13116.213236] LNetError: Protocol error connecting to 192.168.202.128@tcp125 at host 192.168.202.128:988: is it running a compatible version of Lustre? [13116.225994] LNetError: Skipped 29 previous similar messages [13116.684108] LustreError: 463715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782498020 with bad export cookie 1190666679156242408 [13116.695143] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13117.197684] Lustre: server umount lustre-MDT0001 complete [13128.414276] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13130.958649] Key type lgssc unregistered [13131.404709] LNet: 467611:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13131.414695] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13132.450218] LNet: Removed LNI 192.168.202.128@tcp [13132.462877] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13133.472269] LNet: Removed LNI 192.168.202.128@tcp100 [13133.477935] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13133.484318] LNet: Removed LNI 192.168.202.128@tcp101 [13134.496488] LNet: Removed LNI 192.168.202.128@tcp102 [13135.521170] LNet: Removed LNI 192.168.202.128@tcp103 [13135.541854] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13135.556399] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13135.573737] LNet: Removed LNI 192.168.202.128@tcp104 [13136.614072] LNet: Removed LNI 192.168.202.128@tcp105 [13137.632805] LNet: Removed LNI 192.168.202.128@tcp106 [13138.656218] LNet: Removed LNI 192.168.202.128@tcp107 [13139.680803] LNet: Removed LNI 192.168.202.128@tcp108 [13139.683591] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13139.689354] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13139.699355] LNet: Removed LNI 192.168.202.128@tcp109 [13140.768967] LNet: Removed LNI 192.168.202.128@tcp110 [13141.792612] LNet: Removed LNI 192.168.202.128@tcp111 [13142.880444] LNet: Removed LNI 192.168.202.128@tcp112 [13143.904719] LNet: Removed LNI 192.168.202.128@tcp113 [13144.928379] LNet: Removed LNI 192.168.202.128@tcp114 [13145.953188] LNet: Removed LNI 192.168.202.128@tcp115 [13146.980352] LNet: Removed LNI 192.168.202.128@tcp116 [13148.064788] LNet: Removed LNI 192.168.202.128@tcp117 [13148.066965] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13148.094028] LNetError: 467611:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13148.118104] LNet: Removed LNI 192.168.202.128@tcp118 [13149.153527] LNet: Removed LNI 192.168.202.128@tcp119 [13150.176294] LNet: Removed LNI 192.168.202.128@tcp120 [13151.201746] LNet: Removed LNI 192.168.202.128@tcp121 [13152.288237] LNet: Removed LNI 192.168.202.128@tcp122 [13153.376161] LNet: Removed LNI 192.168.202.128@tcp123 [13154.400865] LNet: Removed LNI 192.168.202.128@tcp124 [13155.424395] LNet: Removed LNI 192.168.202.128@tcp125 [13155.427166] LNet: Removed LNI 192.168.202.128@tcp126 [13156.448535] LNet: Removed LNI 192.168.202.128@tcp127 [13157.473224] LNet: Removed LNI 192.168.202.128@tcp128 [13158.497079] LNet: Removed LNI 192.168.202.128@tcp129 [13158.499237] LNet: Removed LNI 192.168.202.128@tcp130 [13158.504303] LNet: Removed LNI 192.168.202.128@tcp131 [13158.510502] LNet: Removed LNI 192.168.202.128@tcp132 [13158.516768] LNet: Removed LNI 192.168.202.128@tcp133 [13158.528732] LNet: Removed LNI 192.168.202.128@tcp134 [13158.547078] LNet: Removed LNI 192.168.202.128@tcp135 [13159.469203] Key type .llcrypt unregistered [13159.471465] Key type ._llcrypt unregistered [13183.521503] Key type ._llcrypt registered [13183.525772] Key type .llcrypt registered [13183.700864] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [13198.268949] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13199.370235] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13199.492853] alg: No test for adler32 (adler32-zlib) [13200.541767] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [13200.888281] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [13202.632167] Key type lgssc registered [13203.699880] Lustre: Echo OBD driver; http://www.lustre.org/ [13234.796031] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:22:18 (1782498138) [13252.823149] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13255.165989] Key type lgssc unregistered [13255.493656] LNet: 471702:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13255.512853] LNetError: 471702:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13255.552904] LNet: Removed LNI 192.168.202.128@tcp [13256.385736] Key type .llcrypt unregistered [13256.387273] Key type ._llcrypt unregistered [13272.432784] Key type ._llcrypt registered [13272.435037] Key type .llcrypt registered [13272.528773] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13273.975254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13274.001236] alg: No test for adler32 (adler32-zlib) [13274.997658] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [13275.169392] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [13276.784296] Key type lgssc registered [13277.983345] Lustre: Echo OBD driver; http://www.lustre.org/ [13287.449150] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13287.473726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13288.731632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13288.788641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13288.858568] Lustre: lustre-MDT0000: new disk, initializing [13288.942693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13288.958622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13294.009418] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13304.881901] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13304.953991] Lustre: 472913:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13305.001453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13305.007347] Lustre: Skipped 1 previous similar message [13305.074550] Lustre: lustre-MDT0001: new disk, initializing [13305.155142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13305.196689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13305.215663] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13309.268664] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13317.633833] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13322.785863] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13328.871143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13329.182454] Lustre: lustre-OST0000: new disk, initializing [13329.191599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13329.202543] Lustre: 474886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13329.339265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13335.543794] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13338.706177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13338.712233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13338.811324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13343.447891] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13346.600430] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing lnet_if_list [13351.957556] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13352.013053] LNet: Added LNI 192.168.202.128@tcp10 [8/256/0/180] [13357.419633] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13357.449995] LNet: Added LNI 192.168.202.128@tcp11 [8/256/0/180] [13361.641704] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13361.661912] LNet: Added LNI 192.168.202.128@tcp12 [8/256/0/180] [13366.187859] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13366.207885] LNet: Added LNI 192.168.202.128@tcp13 [8/256/0/180] [13371.583904] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13371.619829] LNet: Added LNI 192.168.202.128@tcp14 [8/256/0/180] [13376.152596] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13376.184975] LNet: Added LNI 192.168.202.128@tcp15 [8/256/0/180] [13380.837635] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13380.855076] LNet: Added LNI 192.168.202.128@tcp16 [8/256/0/180] [13385.546713] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13385.583853] LNet: Added LNI 192.168.202.128@tcp17 [8/256/0/180] [13390.422123] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13390.460389] LNet: Added LNI 192.168.202.128@tcp18 [8/256/0/180] [13395.211890] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13395.237077] LNet: Added LNI 192.168.202.128@tcp19 [8/256/0/180] [13400.451184] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13400.471957] LNet: Added LNI 192.168.202.128@tcp20 [8/256/0/180] [13405.139249] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13405.169352] LNet: Added LNI 192.168.202.128@tcp21 [8/256/0/180] [13409.945155] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13409.977115] LNet: Added LNI 192.168.202.128@tcp22 [8/256/0/180] [13415.097418] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13415.122695] LNet: Added LNI 192.168.202.128@tcp23 [8/256/0/180] [13420.241772] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13420.277496] LNet: Added LNI 192.168.202.128@tcp24 [8/256/0/180] [13425.686030] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13425.711781] LNet: Added LNI 192.168.202.128@tcp25 [8/256/0/180] [13431.594983] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13431.615893] LNet: Added LNI 192.168.202.128@tcp26 [8/256/0/180] [13438.561662] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13438.609695] LNet: Added LNI 192.168.202.128@tcp27 [8/256/0/180] [13444.498980] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13444.540292] LNet: Added LNI 192.168.202.128@tcp28 [8/256/0/180] [13454.135500] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13454.181066] LNet: Added LNI 192.168.202.128@tcp29 [8/256/0/180] [13462.753613] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13462.837292] LNet: Added LNI 192.168.202.128@tcp30 [8/256/0/180] [13469.238240] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13469.262067] LNet: Added LNI 192.168.202.128@tcp31 [8/256/0/180] [13477.053510] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13477.097197] LNet: Added LNI 192.168.202.128@tcp32 [8/256/0/180] [13485.463263] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13485.511460] LNet: Added LNI 192.168.202.128@tcp33 [8/256/0/180] [13491.398273] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13491.421670] LNet: Added LNI 192.168.202.128@tcp34 [8/256/0/180] [13497.251660] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13497.276966] LNet: Added LNI 192.168.202.128@tcp35 [8/256/0/180] [13504.070304] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13504.098107] LNet: Added LNI 192.168.202.128@tcp36 [8/256/0/180] [13510.964797] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13510.988871] LNet: Added LNI 192.168.202.128@tcp37 [8/256/0/180] [13517.427921] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13517.489175] LNet: Added LNI 192.168.202.128@tcp38 [8/256/0/180] [13524.262450] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13524.279736] LNet: Added LNI 192.168.202.128@tcp39 [8/256/0/180] [13530.693329] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13530.730275] LNet: Added LNI 192.168.202.128@tcp40 [8/256/0/180] [13538.336654] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13538.375501] LNet: Added LNI 192.168.202.128@tcp41 [8/256/0/180] [13545.858615] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13545.894194] LNet: Added LNI 192.168.202.128@tcp42 [8/256/0/180] [13553.042668] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13553.069407] LNet: Added LNI 192.168.202.128@tcp43 [8/256/0/180] [13559.400513] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13559.431392] LNet: Added LNI 192.168.202.128@tcp44 [8/256/0/180] [13566.514382] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13566.547982] LNet: Added LNI 192.168.202.128@tcp45 [8/256/0/180] [13573.952634] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13573.966667] LNet: Added LNI 192.168.202.128@tcp46 [8/256/0/180] [13581.345746] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13581.370839] LNet: Added LNI 192.168.202.128@tcp47 [8/256/0/180] [13589.394730] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13589.429324] LNet: Added LNI 192.168.202.128@tcp48 [8/256/0/180] [13597.009224] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13597.055479] LNet: Added LNI 192.168.202.128@tcp49 [8/256/0/180] [13604.901310] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13604.938039] LNet: Added LNI 192.168.202.128@tcp50 [8/256/0/180] [13625.828286] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13625.832068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13625.866703] Lustre: Skipped 2 previous similar messages [13625.906387] Lustre: Skipped 2 previous similar messages [13629.858808] Lustre: server umount lustre-MDT0000 complete [13634.305430] LustreError: 472908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782498538 with bad export cookie 8261562394123645327 [13634.315613] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13634.327210] LustreError: 472908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13635.065740] Lustre: server umount lustre-MDT0001 complete [13651.424123] Lustre: 472186:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782498539/real 1782498539] req@00000000a899640c x1869084809656960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782498555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13651.437884] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13653.594601] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13654.560706] Lustre: 472186:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782498542/real 1782498542] req@00000000da008d22 x1869084809657088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782498558 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13656.544826] Lustre: 472188:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782498544/real 1782498544] req@0000000047f2da6d x1869084809657216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782498560 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13660.655626] Lustre: 472189:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782498548/real 1782498548] req@000000004dfe4775 x1869084809657472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782498564 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13663.736642] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13666.787458] Lustre: 472186:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782498554/real 1782498554] req@000000000850a45a x1869084809657600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782498570 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13680.227237] LustreError: 472185:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004d888557 x1869084809658368/t0(0) o250->MGC192.168.202.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13680.687266] LustreError: 474878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13680.706484] LustreError: 474878:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13680.746112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13681.765566] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13681.771454] LustreError: 474879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13682.785980] LustreError: 474878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13684.993775] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13688.300330] LustreError: 474879:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13693.412818] LustreError: 474878:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13694.930290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13695.460980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13696.495092] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13700.904905] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13709.474610] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13715.032560] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13736.418570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13736.422566] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13736.444504] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13736.931881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13737.446247] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13737.461983] Lustre: Skipped 2 previous similar messages [13742.563558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13742.569756] Lustre: Skipped 1 previous similar message [13747.168172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13747.431624] Lustre: server umount lustre-OST0000 complete [13751.777936] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13751.783881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13751.791475] Lustre: Skipped 1 previous similar message [13751.795873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13757.463950] Lustre: server umount lustre-MDT0000 complete [13757.925156] LustreError: 499130:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13757.978361] LustreError: 499130:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13761.500553] LustreError: 473711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782498665 with bad export cookie 8261562394123647910 [13761.510054] LustreError: 473711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13761.511591] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13761.824215] Lustre: server umount lustre-MDT0001 complete [13774.367478] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13778.138341] Key type lgssc unregistered [13778.896608] LNet: 501907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13778.903768] LNetError: 501907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13778.919912] LNet: Removed LNI 192.168.202.128@tcp [13778.921635] LNet: Removed LNI 192.168.202.128@tcp10 [13778.931959] LNet: Removed LNI 192.168.202.128@tcp11 [13778.937780] LNet: Removed LNI 192.168.202.128@tcp12 [13778.942310] LNet: Removed LNI 192.168.202.128@tcp13 [13778.947233] LNet: Removed LNI 192.168.202.128@tcp14 [13778.957295] LNet: Removed LNI 192.168.202.128@tcp15 [13778.960637] LNet: Removed LNI 192.168.202.128@tcp16 [13778.971384] LNet: Removed LNI 192.168.202.128@tcp17 [13778.975822] LNet: Removed LNI 192.168.202.128@tcp18 [13778.990212] LNet: Removed LNI 192.168.202.128@tcp19 [13778.999312] LNet: Removed LNI 192.168.202.128@tcp20 [13779.003913] LNet: Removed LNI 192.168.202.128@tcp21 [13779.013898] LNet: Removed LNI 192.168.202.128@tcp22 [13779.021336] LNet: Removed LNI 192.168.202.128@tcp23 [13779.033506] LNet: Removed LNI 192.168.202.128@tcp24 [13779.039256] LNet: Removed LNI 192.168.202.128@tcp25 [13779.041953] LNet: Removed LNI 192.168.202.128@tcp26 [13779.047895] LNet: Removed LNI 192.168.202.128@tcp27 [13779.053311] LNet: Removed LNI 192.168.202.128@tcp28 [13779.058642] LNet: Removed LNI 192.168.202.128@tcp29 [13779.066259] LNet: Removed LNI 192.168.202.128@tcp30 [13779.072224] LNet: Removed LNI 192.168.202.128@tcp31 [13779.076668] LNet: Removed LNI 192.168.202.128@tcp32 [13779.082598] LNet: Removed LNI 192.168.202.128@tcp33 [13779.097787] LNet: Removed LNI 192.168.202.128@tcp34 [13779.099852] LNet: Removed LNI 192.168.202.128@tcp35 [13779.104265] LNet: Removed LNI 192.168.202.128@tcp36 [13779.115482] LNet: Removed LNI 192.168.202.128@tcp37 [13779.120454] LNet: Removed LNI 192.168.202.128@tcp38 [13779.124325] LNet: Removed LNI 192.168.202.128@tcp39 [13779.129826] LNet: Removed LNI 192.168.202.128@tcp40 [13779.143414] LNet: Removed LNI 192.168.202.128@tcp41 [13779.148312] LNet: Removed LNI 192.168.202.128@tcp42 [13779.154609] LNet: Removed LNI 192.168.202.128@tcp43 [13779.161405] LNet: Removed LNI 192.168.202.128@tcp44 [13779.178396] LNet: Removed LNI 192.168.202.128@tcp45 [13779.180184] LNet: Removed LNI 192.168.202.128@tcp46 [13779.189266] LNet: Removed LNI 192.168.202.128@tcp47 [13779.192995] LNet: Removed LNI 192.168.202.128@tcp48 [13779.206195] LNet: Removed LNI 192.168.202.128@tcp49 [13779.237980] LNet: Removed LNI 192.168.202.128@tcp50 [13780.197669] Key type .llcrypt unregistered [13780.201579] Key type ._llcrypt unregistered [13802.078227] Key type ._llcrypt registered [13802.080457] Key type .llcrypt registered [13802.175742] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [13817.834708] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13819.901830] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13819.947526] alg: No test for adler32 (adler32-zlib) [13821.178092] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [13821.462434] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [13823.169303] Key type lgssc registered [13824.853506] Lustre: Echo OBD driver; http://www.lustre.org/ [13859.686671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13859.724136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13860.918825] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13860.943084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13861.002801] Lustre: lustre-MDT0000: new disk, initializing [13861.060115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13861.076904] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13865.316447] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13874.806337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13874.943770] Lustre: 505104:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13875.004097] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13875.008811] Lustre: Skipped 1 previous similar message [13875.123236] Lustre: lustre-MDT0001: new disk, initializing [13875.253848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13875.291967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13875.311838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13880.110871] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13888.648267] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13893.188209] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13899.865680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13900.041761] Lustre: *** cfs_fail_loc=90f, val=0*** [13900.104804] Lustre: lustre-OST0000: new disk, initializing [13900.108800] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13900.114688] Lustre: 507119:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13900.222831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13905.723498] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [13908.080473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13908.097463] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13908.199592] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13915.404664] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13923.812818] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13923.815020] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13923.818318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13923.832924] LustreError: Skipped 1 previous similar message [13923.864739] Lustre: Skipped 1 previous similar message [13928.886676] Lustre: server umount lustre-OST0000 complete [13934.054633] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13934.061148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13934.079745] Lustre: Skipped 2 previous similar messages [13938.825979] Lustre: server umount lustre-MDT0000 complete [13942.586392] LustreError: 505095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782498846 with bad export cookie 8104760271129350974 [13942.597847] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13942.604412] LustreError: 505095:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13942.888679] Lustre: server umount lustre-MDT0001 complete [13956.662367] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [13960.269544] Key type lgssc unregistered [13960.618242] LNet: 509027:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13960.624161] LNetError: 509027:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13960.648973] LNet: Removed LNI 192.168.202.128@tcp [13961.466759] Key type .llcrypt unregistered [13961.468884] Key type ._llcrypt unregistered [13980.983865] Key type ._llcrypt registered [13980.986175] Key type .llcrypt registered [13981.106468] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [13982.180791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13982.344092] alg: No test for adler32 (adler32-zlib) [13983.303657] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [13983.516217] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [13985.244087] Key type lgssc registered [13986.505636] Lustre: Echo OBD driver; http://www.lustre.org/ [13993.302285] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13993.317742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13994.826992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13999.067639] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14005.225239] LustreError: 510300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14005.252326] LustreError: 510300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14007.837784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14008.289866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14013.657770] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14021.976088] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14027.123106] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14032.688624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14032.814138] LustreError: 512122:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14033.062915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14038.247605] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14045.358614] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14056.593762] loop0: detected capacity change from 0 to 204800000 [14056.776947] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14056.788464] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14056.797299] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14063.034949] loop0: detected capacity change from 0 to 204800000 [14063.289711] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14063.310526] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14063.348723] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14068.556312] loop0: detected capacity change from 0 to 204800000 [14073.465301] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14073.783963] Lustre: fs15246-OST0000: new disk, initializing [14073.788042] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14073.798341] Lustre: 513896:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14073.865500] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14082.530975] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14091.560923] loop1: detected capacity change from 0 to 204800000 [14097.206442] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14097.316506] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14097.351111] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14097.401525] Lustre: fs15246-MDT0000: new disk, initializing [14097.525610] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14097.547165] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14097.623195] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14098.657234] Lustre: 515018:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000046a74362 x1869085552982400/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782499018 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14102.144188] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14107.748345] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14107.756824] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14107.771684] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14125.541534] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14125.561897] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14129.649976] Lustre: server umount fs15246-MDT0000 complete [14130.663207] LustreError: 510295:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14135.785985] LustreError: 511257:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14138.936987] Lustre: server umount fs15246-OST0000 complete [14151.147739] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14151.153133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14151.156274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14151.165205] Lustre: Skipped 1 previous similar message [14156.259292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14156.275337] Lustre: Skipped 2 previous similar messages [14156.637930] Lustre: server umount lustre-OST0000 complete [14161.378444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14161.388548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14166.573632] LustreError: 511256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14166.749655] Lustre: server umount lustre-MDT0000 complete [14170.222269] LustreError: 510280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782499073 with bad export cookie 1410811921709658859 [14170.224378] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14170.232573] LustreError: 510280:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14170.623817] Lustre: server umount lustre-MDT0001 complete [14182.941163] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [14185.965333] Key type lgssc unregistered [14186.313397] LNet: 518113:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14186.329315] LNetError: 518113:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14186.354757] LNet: Removed LNI 192.168.202.128@tcp [14187.275179] Key type .llcrypt unregistered [14187.281310] Key type ._llcrypt unregistered [14206.930289] Key type ._llcrypt registered [14206.936748] Key type .llcrypt registered [14207.043525] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [14208.398207] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14208.695677] alg: No test for adler32 (adler32-zlib) [14209.925563] Lustre: Lustre: Build Version: 2.17.54_4_g4cd08f3 [14210.203785] LNet: Added LNI 192.168.202.128@tcp [8/256/0/180] [14211.896565] Key type lgssc registered [14213.388830] Lustre: Echo OBD driver; http://www.lustre.org/ [14255.116962] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [14264.640501] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [14304.761670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14304.817205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14306.123945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14306.162433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14306.248453] Lustre: lustre-MDT0000: new disk, initializing [14306.339866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14306.357111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14312.901216] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14324.659990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14324.813389] Lustre: 522581:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14324.886341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14324.893181] Lustre: Skipped 1 previous similar message [14324.996708] Lustre: lustre-MDT0001: new disk, initializing [14325.127559] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14325.180452] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14325.195497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14329.190281] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14340.180893] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14347.086963] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14355.371732] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14355.896620] Lustre: lustre-OST0000: new disk, initializing [14355.903397] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14355.924031] Lustre: 524552:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14356.126077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14362.672095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14362.678353] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14362.808172] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14363.462820] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_default_debug -1 all [14373.632854] Lustre: DEBUG MARKER: oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14379.780611] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14380.058731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14385.271495] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14385.535726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14388.200174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14388.209539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14388.231819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14388.722701] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14388.728197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14388.746973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14393.830360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14393.846283] Lustre: Skipped 1 previous similar message [14398.968103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14398.980627] Lustre: Skipped 1 previous similar message [14402.019824] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14402.257474] Lustre: server umount lustre-OST0000 complete [14407.137611] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14407.146853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14407.160749] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14409.187463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14412.673178] Lustre: server umount lustre-MDT0000 complete [14416.955337] LustreError: 522574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782499320 with bad export cookie 1332893077940770076 [14416.962326] LustreError: MGC192.168.202.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14416.976718] LustreError: 522574:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14417.369675] Lustre: server umount lustre-MDT0001 complete [14450.236203] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing set_hostid [14461.633777] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing load_modules_local [14495.147967] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14218 sec ========= 14:43:18 (1782499398) [14496.973400] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:43:20 (1782499400) === [14501.995445] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:43:24 (1782499404) === [14551.343765] Lustre: DEBUG MARKER: oleg228-server.virtnet: executing unload_modules_local [14556.253664] Key type lgssc unregistered [14556.776150] LNet: 531202:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14556.792423] LNetError: 531202:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14556.805851] LNet: Removed LNI 192.168.202.128@tcp [14557.991158] Key type .llcrypt unregistered [14557.992483] Key type ._llcrypt unregistered