[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 468858232 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003090] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005015] kvm-guest: setup PV IPIs [ 0.008615] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010014] pid_max: default: 32768 minimum: 301 [ 0.011150] LSM: Security Framework initializing [ 0.012055] Yama: becoming mindful. [ 0.013058] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022448] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027183] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028159] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029119] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030114] *** VALIDATE tmpfs *** [ 0.031376] *** VALIDATE proc *** [ 0.033240] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.036005] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037163] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039031] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.043195] debug: unmapping init [mem 0xffffffff88259000-0xffffffff88260fff] [ 0.045874] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046605] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047025] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049009] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051015] ... max period: 00007fffffffffff [ 0.052013] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054310] rcu: Hierarchical SRCU implementation. [ 0.056415] smp: Bringing up secondary CPUs ... [ 0.057611] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.061319] smp: Brought up 1 node, 4 CPUs [ 0.063012] smpboot: Max logical packages: 1 [ 0.064018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.130067] node 0 deferred pages initialised in 63ms [ 0.134176] devtmpfs: initialized [ 0.136350] x86/mm: Memory block size: 128MB [ 0.139962] gcov: version magic: 0x41383552 [ 0.143186] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.144088] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.146375] pinctrl core: initialized pinctrl subsystem [ 0.147217] [ 0.147854] ************************************************************* [ 0.150014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.152012] ** ** [ 0.155012] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.156000] ** ** [ 0.157011] ** This means that this kernel is built to expose internal ** [ 0.159014] ** IOMMU data structures, which may compromise security on ** [ 0.162014] ** your system. ** [ 0.164011] ** ** [ 0.166014] ** If you see this message and you are not debugging the ** [ 0.168066] ** kernel, report this immediately to your vendor! ** [ 0.169009] ** ** [ 0.171012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.174022] ************************************************************* [ 0.176831] NET: Registered protocol family 16 [ 0.179411] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.182098] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.184073] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.189096] cpuidle: using governor menu [ 0.191925] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.194910] PCI: Using configuration type 1 for base access [ 0.197129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.206204] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.208031] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.212073] cryptd: max_cpu_qlen set to 1000 [ 0.215223] ACPI: Added _OSI(Module Device) [ 0.217016] ACPI: Added _OSI(Processor Device) [ 0.218012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.220015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.224163] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.230513] ACPI: Interpreter enabled [ 0.232058] ACPI: PM: (supports S0 S3 S4 S5) [ 0.233011] ACPI: Using IOAPIC for interrupt routing [ 0.235126] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.237273] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.247064] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.250043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.252019] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.255081] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.260533] acpiphp: Slot [2] registered [ 0.263137] acpiphp: Slot [5] registered [ 0.264100] acpiphp: Slot [6] registered [ 0.265111] acpiphp: Slot [7] registered [ 0.267113] acpiphp: Slot [8] registered [ 0.268100] acpiphp: Slot [9] registered [ 0.270124] acpiphp: Slot [10] registered [ 0.271083] acpiphp: Slot [3] registered [ 0.272082] acpiphp: Slot [4] registered [ 0.274089] acpiphp: Slot [11] registered [ 0.275084] acpiphp: Slot [12] registered [ 0.276092] acpiphp: Slot [13] registered [ 0.278089] acpiphp: Slot [14] registered [ 0.279113] acpiphp: Slot [15] registered [ 0.280081] acpiphp: Slot [16] registered [ 0.281089] acpiphp: Slot [17] registered [ 0.283088] acpiphp: Slot [18] registered [ 0.284178] acpiphp: Slot [19] registered [ 0.286105] acpiphp: Slot [20] registered [ 0.287130] acpiphp: Slot [21] registered [ 0.288103] acpiphp: Slot [22] registered [ 0.290094] acpiphp: Slot [23] registered [ 0.291101] acpiphp: Slot [24] registered [ 0.293088] acpiphp: Slot [25] registered [ 0.294100] acpiphp: Slot [26] registered [ 0.296091] acpiphp: Slot [27] registered [ 0.297102] acpiphp: Slot [28] registered [ 0.298107] acpiphp: Slot [29] registered [ 0.300081] acpiphp: Slot [30] registered [ 0.301104] acpiphp: Slot [31] registered [ 0.302047] PCI host bridge to bus 0000:00 [ 0.303016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305020] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307020] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.308016] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.310020] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.313050] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.315188] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.317980] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.321204] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.330018] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.334057] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.337019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.340018] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.342018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.345360] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.348814] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.351033] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.354813] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.360012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.374023] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.379827] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.384961] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.397015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.406016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.421028] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.438096] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.449014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.459014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.489016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.501571] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.509015] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.517016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.544019] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.555321] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.561026] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.567020] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.587017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.596695] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.604024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.610019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.643016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.655846] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.669020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.677015] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.699016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.710012] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.712488] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.715459] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.717375] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.719211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.723225] iommu: Default domain type: Passthrough [ 0.725470] SCSI subsystem initialized [ 0.727196] ACPI: bus type USB registered [ 0.729114] usbcore: registered new interface driver usbfs [ 0.731095] usbcore: registered new interface driver hub [ 0.733117] usbcore: registered new device driver usb [ 0.735140] pps_core: LinuxPPS API ver. 1 registered [ 0.737011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.739094] PTP clock support registered [ 0.742048] EDAC MC: Ver: 3.0.0 [ 0.743534] PCI: Using ACPI for IRQ routing [ 0.745149] NetLabel: Initializing [ 0.746000] NetLabel: domain hash size = 128 [ 0.748016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.750086] NetLabel: unlabeled traffic allowed by default [ 0.752170] vgaarb: loaded [ 0.754278] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.756011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.761725] clocksource: Switched to clocksource kvm-clock [ 0.868519] VFS: Disk quotas dquot_6.6.0 [ 0.870107] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.872817] *** VALIDATE ramfs *** [ 0.874079] *** VALIDATE hugetlbfs *** [ 0.875840] pnp: PnP ACPI init [ 0.878148] pnp: PnP ACPI: found 6 devices [ 0.901089] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.904445] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.906796] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.909035] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.911467] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.913930] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.916883] NET: Registered protocol family 2 [ 0.919578] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.924421] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.927758] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.933357] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.936856] TCP: Hash tables configured (established 65536 bind 65536) [ 0.939989] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.943428] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946185] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949060] NET: Registered protocol family 1 [ 0.951748] RPC: Registered named UNIX socket transport module. [ 0.954115] RPC: Registered udp transport module. [ 0.956058] RPC: Registered tcp transport module. [ 0.957895] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.960355] NET: Registered protocol family 44 [ 0.962153] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.964233] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.966562] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.969122] PCI: CLS 0 bytes, default 64 [ 0.970736] Unpacking initramfs... [ 2.367258] debug: unmapping init [mem 0xffff99173cc54000-0xffff99173ffbffff] [ 2.371421] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.373799] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.376854] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.878895] Initialise system trusted keyrings [ 2.880815] Key type blacklist registered [ 2.882975] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.894490] zbud: loaded [ 2.898250] *** VALIDATE nfs *** [ 2.899521] *** VALIDATE nfs4 *** [ 2.901030] pstore: using deflate compression [ 2.907453] Platform Keyring initialized [ 3.027616] NET: Registered protocol family 38 [ 3.029610] Key type asymmetric registered [ 3.030939] Asymmetric key parser 'x509' registered [ 3.033459] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.037048] io scheduler mq-deadline registered [ 3.038835] io scheduler kyber registered [ 3.040413] io scheduler bfq registered [ 3.042326] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.045383] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.048271] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.050857] ACPI: Power Button [PWRF] [ 3.056157] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.063186] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.076127] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.082900] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.098685] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.124489] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.154265] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.159848] Non-volatile memory driver v1.3 [ 3.161545] Linux agpgart interface v0.103 [ 3.195232] virtio_blk virtio1: [vda] 146648 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.197913] vda: detected capacity change from 0 to 75083776 [ 3.210696] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.213876] vdb: detected capacity change from 0 to 1073741824 [ 3.227956] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.230126] vdc: detected capacity change from 0 to 2621440000 [ 3.243932] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.246756] vdd: detected capacity change from 0 to 2621440000 [ 3.260912] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.264572] vde: detected capacity change from 0 to 4294967296 [ 3.280494] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.283735] vdf: detected capacity change from 0 to 4294967296 [ 3.296417] libphy: Fixed MDIO Bus: probed [ 3.303390] usbcore: registered new interface driver usbserial_generic [ 3.306097] usbserial: USB Serial support registered for generic [ 3.308568] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.313397] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.315288] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.317663] mousedev: PS/2 mouse device common for all mice [ 3.321324] rtc_cmos 00:05: RTC can wake from S4 [ 3.325465] rtc_cmos 00:05: registered as rtc0 [ 3.326804] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.328662] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.334265] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.335038] intel_pstate: CPU model not supported [ 3.339954] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.344204] hid: raw HID events driver (C) Jiri Kosina [ 3.346178] usbcore: registered new interface driver usbhid [ 3.348068] usbhid: USB HID core driver [ 3.349662] drop_monitor: Initializing network drop monitor service [ 3.351290] Initializing XFRM netlink socket [ 3.352911] NET: Registered protocol family 10 [ 3.355122] Segment Routing with IPv6 [ 3.356258] NET: Registered protocol family 17 [ 3.357971] mpls_gso: MPLS GSO support [ 3.365227] RAS: Correctable Errors collector initialized. [ 3.367555] AVX version of gcm_enc/dec engaged. [ 3.369378] AES CTR mode by8 optimization enabled [ 3.439859] sched_clock: Marking stable (3439835779, 0)->(4365407710, -925571931) [ 3.443636] registered taskstats version 1 [ 3.445567] Loading compiled-in X.509 certificates [ 3.447617] zswap: loaded using pool lzo/zbud [ 3.470379] Key type big_key registered [ 3.482487] Key type encrypted registered [ 3.483773] ima: No TPM chip found, activating TPM-bypass! [ 3.485692] ima: Allocated hash algorithm: sha1 [ 3.488525] ima: No architecture policies found [ 3.490205] evm: Initialising EVM extended attributes: [ 3.491943] evm: security.selinux [ 3.493013] evm: security.ima [ 3.493820] evm: security.capability [ 3.494842] evm: HMAC attrs: 0x1 [ 3.497147] rtc_cmos 00:05: setting system clock to 2026-09-07 10:01:00 UTC (1788775260) [ 3.503201] debug: unmapping init [mem 0xffffffff89203000-0xffffffff893fffff] [ 3.506174] debug: unmapping init [mem 0xffffffff87f82000-0xffffffff88258fff] [ 3.519173] Write protecting the kernel read-only data: 28672k [ 3.522177] debug: unmapping init [mem 0xffffffff86603000-0xffffffff867fffff] [ 3.524665] debug: unmapping init [mem 0xffffffff86f14000-0xffffffff86ffffff] [ 3.558974] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.566315] systemd[1]: Detected virtualization kvm. [ 3.568093] systemd[1]: Detected architecture x86-64. [ 3.569668] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.598314] systemd[1]: No hostname configured. [ 3.599776] systemd[1]: Set hostname to . [ 3.601461] random: systemd: uninitialized urandom read (16 bytes read) [ 3.603471] systemd[1]: Initializing machine ID from random generator. [ 3.663704] random: ln: uninitialized urandom read (6 bytes read) [ 3.746884] random: systemd: uninitialized urandom read (16 bytes read) [ 3.749479] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.755422] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.759787] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Started Memstrack Anylazing Service. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.391284] device-mapper: uevent: version 1.0.3 [ 4.394314] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. [ 4.757271] random: fast init done Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.078974] virtio_net virtio0 ens2: renamed from eth0 [ 5.129037] scsi host0: ata_piix [ 5.172673] scsi host1: ata_piix [ 5.174473] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.177188] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.448892] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.000588] random: crng init done [ 10.001829] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.357046] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.462265] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.723892] SELinux: Disabled at runtime. [ 11.786043] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.794390] systemd[1]: Detected virtualization kvm. [ 11.796408] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.291426] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.295220] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.303073] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.307725] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.311780] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.320190] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.329373] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd Root File System. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ 12.491145] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.748852] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.976216] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.005716] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.087536] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.097492] EDAC sbridge: Ver: 1.1.2 [ 14.874238] Key type dns_resolver registered [ 15.173829] NFS: Registering the id_resolver key type [ 15.176562] Key type id_resolver registered [ 15.178756] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg425-server login: [ 55.584445] libcfs: loading out-of-tree module taints kernel. [ 55.642913] Key type ._llcrypt registered [ 55.646586] Key type .llcrypt registered [ 55.744814] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 73.540993] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 75.107194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 75.120130] alg: No test for adler32 (adler32-zlib) [ 76.520388] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 77.376800] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 79.128119] Key type lgssc registered [ 80.772534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 100.245528] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 105.314157] hrtimer: interrupt took 5163946 ns [ 143.301339] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 158.468171] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 158.507824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 159.781958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 159.840555] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 159.986682] Lustre: lustre-MDT0000: new disk, initializing [ 160.094623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 160.114269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 164.636349] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 178.545915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 178.711397] Lustre: 6501:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 178.777624] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 178.783965] Lustre: Skipped 1 previous similar message [ 178.913590] Lustre: lustre-MDT0001: new disk, initializing [ 178.988257] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 179.020989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 179.036237] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 183.392785] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 188.570990] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 200.790896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 201.104551] Lustre: lustre-OST0000: new disk, initializing [ 201.108698] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 201.115110] Lustre: 8439:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 201.194548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 205.873811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 205.885341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 205.935854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 207.719488] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 222.619442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 222.721878] Lustre: lustre-OST0001: new disk, initializing [ 222.725228] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 222.730404] Lustre: 9512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 222.780084] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 228.889429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.905114] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.994822] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 229.210183] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 241.404947] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.502685] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 256.657172] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing check_logdir /tmp/testlogs/ [ 261.689399] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing yml_node [ 266.976416] Lustre: DEBUG MARKER: Client: 2.17.57.103 [ 270.197775] Lustre: DEBUG MARKER: MDS: 2.17.57.103 [ 273.115435] Lustre: DEBUG MARKER: OSS: 2.17.57.103 [ 275.023151] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 06:05:30 EDT 2026 [ 292.850276] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 294.454603] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 312.293980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 312.313420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 312.357716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.872122] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.876666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.914716] Lustre: Skipped 2 previous similar messages [ 315.919380] Lustre: Skipped 2 previous similar messages [ 318.728804] Lustre: server umount lustre-MDT0000 complete [ 322.910935] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775579 with bad export cookie 16145419021966880594 [ 322.918144] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 322.926237] LustreError: 6495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 326.117180] LustreError: 6507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.118704] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.124583] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.124592] Lustre: Skipped 1 previous similar message [ 326.138658] LustreError: 6507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 326.187691] Lustre: Skipped 1 previous similar message [ 327.135804] LustreError: 8657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 327.159331] LustreError: 8657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 332.256850] LustreError: 11077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 332.257828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 332.281717] LustreError: 11077:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 332.287192] Lustre: Skipped 4 previous similar messages [ 337.379430] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 337.382302] LustreError: 6507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 337.383153] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 337.414785] LustreError: 6507:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 337.659325] Lustre: server umount lustre-MDT0001 complete [ 341.489934] Lustre: server umount lustre-OST0000 complete [ 345.140070] Lustre: server umount lustre-OST0001 complete [ 350.880910] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 360.595976] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 399.841782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 400.070285] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 400.117317] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 400.193233] Lustre: lustre-MDT0000: new disk, initializing [ 400.247772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 400.267988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 404.124261] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 415.362735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 415.475642] Lustre: 15069:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 415.546960] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 415.555672] Lustre: Skipped 1 previous similar message [ 415.636937] Lustre: lustre-MDT0001: new disk, initializing [ 415.790433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 415.830436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 415.842096] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 419.911794] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 430.743169] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 437.884866] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 447.566962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 448.009409] Lustre: lustre-OST0000: new disk, initializing [ 448.014455] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 448.023467] Lustre: 17041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 448.160793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.701720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 449.718955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 449.911355] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.690789] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 465.630464] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 470.821288] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 471.111455] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.663751] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.834948] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 480.736578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 480.743675] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 480.752766] LustreError: Skipped 1 previous similar message [ 480.756736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 480.756747] Lustre: Skipped 1 previous similar message [ 480.793494] Lustre: Skipped 1 previous similar message [ 491.999174] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 492.193682] Lustre: server umount lustre-OST0000 complete [ 497.632427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 497.641767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 497.654458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 497.656204] Lustre: Skipped 5 previous similar messages [ 502.594892] Lustre: server umount lustre-MDT0000 complete [ 505.614856] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775762 with bad export cookie 16145419021966882127 [ 505.617917] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 505.626730] LustreError: 15060:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 505.902829] Lustre: server umount lustre-MDT0001 complete [ 510.053669] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 06:09:25 (1788775765) [ 513.101306] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 514.982355] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 06:09:30 (1788775770) [ 522.499240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.001657] LustreError: 19452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 523.106950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.804867] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 533.476809] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.500468] LustreError: 19453:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 537.327244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 542.351794] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 550.005781] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 556.768932] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 565.260094] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 574.109502] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 582.857300] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 592.358292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 592.368457] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 592.378956] Lustre: Skipped 1 previous similar message [ 592.380607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 592.380615] Lustre: Skipped 2 previous similar messages [ 596.502431] Lustre: server umount lustre-OST0000 complete [ 614.368763] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 614.659399] Lustre: server umount lustre-MDT0000 complete [ 617.955995] LustreError: 20991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 617.965845] LustreError: 20991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 619.189397] LustreError: 19433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788775876 with bad export cookie 16145419021966883009 [ 619.192849] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 619.201819] LustreError: 19433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 619.720693] Lustre: server umount lustre-MDT0001 complete [ 634.049159] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 636.299227] Key type lgssc unregistered [ 636.616602] LNet: 22985:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 636.623147] LNetError: 22985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 636.657311] LNet: Removed LNI 192.168.204.125@tcp [ 637.295143] Key type .llcrypt unregistered [ 637.299534] Key type ._llcrypt unregistered [ 666.719364] Key type ._llcrypt registered [ 666.721156] Key type .llcrypt registered [ 666.925706] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 667.663390] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 667.757721] alg: No test for adler32 (adler32-zlib) [ 668.762587] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 668.963113] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 670.647201] Key type lgssc registered [ 671.779688] Lustre: Echo OBD driver; http://www.lustre.org/ [ 680.647651] loop: module loaded [ 680.712374] loop0: detected capacity change from 0 to 204800000 [ 680.857381] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 680.862986] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 681.052547] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.449517] loop0: detected capacity change from 0 to 204800000 [ 687.584877] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.609639] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 687.637572] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 693.314022] loop0: detected capacity change from 0 to 204800000 [ 698.852726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 698.877529] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 700.226209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 700.328139] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 700.380283] Lustre: lustre-MDT0000: new disk, initializing [ 700.476185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 700.508540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 704.988379] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 714.406641] loop1: detected capacity change from 0 to 204800000 [ 720.992562] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 721.184723] Lustre: lustre-OST07c6: new disk, initializing [ 721.191326] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 721.197167] Lustre: Skipped 1 previous similar message [ 721.205184] Lustre: 26174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 721.339953] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 728.539970] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 730.241884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 730.260598] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 730.330092] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 734.660368] Lustre: 25650:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 734.671647] Lustre: 25650:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 734.677073] Lustre: 25650:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 734.682337] Lustre: 25650:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 734.689232] Lustre: 25650:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 734.697322] Lustre: 25650:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 749.168892] Lustre: 25157:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 749.197295] Lustre: 25157:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 749.213686] Lustre: 25157:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 749.227547] Lustre: 25157:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 749.244679] Lustre: 25157:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 749.252580] Lustre: 25157:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.906674] Lustre: 25650:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 754.929595] Lustre: 25650:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 754.943769] Lustre: 25650:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 754.959536] Lustre: 25650:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.978055] Lustre: 25650:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 755.001493] Lustre: 25650:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.013245] Lustre: 25650:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 755.026740] Lustre: 25650:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.056266] Lustre: 25650:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 755.069325] Lustre: 25650:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 755.086057] Lustre: 25650:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 755.091100] Lustre: 25650:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.367001] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 758.377273] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 758.388493] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 758.401156] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.413804] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 758.427269] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.434267] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 758.443917] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.449546] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 758.455935] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 758.461551] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 758.467424] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.545513] Lustre: 25650:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 763.562142] Lustre: 25650:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 763.571502] Lustre: 25650:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.580903] Lustre: 25650:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.589238] Lustre: 25650:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.595943] Lustre: 25650:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.603239] Lustre: 25650:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 763.615160] Lustre: 25650:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.621763] Lustre: 25650:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.629806] Lustre: 25650:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.641862] Lustre: 25650:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.653573] Lustre: 25650:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.095970] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 772.105968] Lustre: 25156:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 772.111168] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 772.113707] Lustre: 25156:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.130181] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.143993] Lustre: 25156:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.149031] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 772.153110] Lustre: 25156:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.157035] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.161807] Lustre: 25156:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 772.166055] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.172174] Lustre: 25156:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 827.365704] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 827.372297] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 827.386909] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 829.442219] Lustre: server umount lustre-OST07c6 complete [ 840.095722] Lustre: server umount lustre-MDT0000 complete [ 857.537984] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 860.131905] Key type lgssc unregistered [ 860.443651] LNet: 28728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 860.458724] LNetError: 28728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 860.480960] LNet: Removed LNI 192.168.204.125@tcp [ 861.313280] Key type .llcrypt unregistered [ 861.316879] Key type ._llcrypt unregistered [ 881.104516] Key type ._llcrypt registered [ 881.110880] Key type .llcrypt registered [ 881.263996] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 882.624898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.663085] alg: No test for adler32 (adler32-zlib) [ 883.812342] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 884.194714] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 885.912797] Key type lgssc registered [ 887.082659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 895.100308] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 895.126449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 896.720461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 901.449966] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 906.210181] LustreError: 30008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 906.223769] LustreError: 30008:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 909.334738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 909.665102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 914.036502] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 921.456506] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 926.957703] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 934.376537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 934.727623] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 940.316615] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 941.868800] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 947.311682] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 967.663620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 967.666371] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.677101] LustreError: Skipped 1 previous similar message [ 967.696301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.698596] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 969.698817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 969.703155] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 969.729344] Lustre: Skipped 1 previous similar message [ 971.871805] Lustre: server umount lustre-OST0000 complete [ 976.357544] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 976.367925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 976.399481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.939233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.942518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 979.973734] Lustre: Skipped 1 previous similar message [ 982.516490] Lustre: server umount lustre-MDT0000 complete [ 985.057498] LustreError: 30738:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 985.078018] LustreError: 30738:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 986.544511] LustreError: 29990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776243 with bad export cookie 16597863240531393584 [ 986.561942] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 986.563631] LustreError: 29990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 986.843060] Lustre: server umount lustre-MDT0001 complete [ 998.842911] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1001.922697] Key type lgssc unregistered [ 1002.394375] LNet: 33746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1002.407241] LNetError: 33746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1002.431399] LNet: Removed LNI 192.168.204.125@tcp [ 1003.344507] Key type .llcrypt unregistered [ 1003.347504] Key type ._llcrypt unregistered [ 1024.311701] Key type ._llcrypt registered [ 1024.314936] Key type .llcrypt registered [ 1024.406574] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1025.607411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1025.647440] alg: No test for adler32 (adler32-zlib) [ 1026.776767] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1027.110878] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1028.831355] Key type lgssc registered [ 1030.224565] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1041.399528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1041.438164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.028743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1047.987934] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1052.136554] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1052.163851] LustreError: 35014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1056.859526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1057.444912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1061.767768] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1070.334669] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1076.796720] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1084.728286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1085.507622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1088.637253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1092.529240] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1101.270335] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1119.722737] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1119.724047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1119.739079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1119.768813] Lustre: Skipped 1 previous similar message [ 1121.026104] Lustre: server umount lustre-OST0000 complete [ 1124.833127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1124.843731] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1124.846640] Lustre: Skipped 1 previous similar message [ 1124.862714] Lustre: Skipped 2 previous similar messages [ 1129.956844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1129.960139] Lustre: Skipped 1 previous similar message [ 1131.004079] Lustre: server umount lustre-MDT0000 complete [ 1135.073392] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1135.100836] LustreError: 35013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1135.158717] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776392 with bad export cookie 8932215680075831534 [ 1135.168263] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1135.178164] LustreError: 34993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1135.917546] Lustre: server umount lustre-MDT0001 complete [ 1146.817190] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1149.460690] Key type lgssc unregistered [ 1149.742432] LNet: 38566:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1149.751574] LNetError: 38566:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1149.774127] LNet: Removed LNI 192.168.204.125@tcp [ 1150.449443] Key type .llcrypt unregistered [ 1150.453191] Key type ._llcrypt unregistered [ 1179.992775] Key type ._llcrypt registered [ 1179.995233] Key type .llcrypt registered [ 1180.103503] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1181.391180] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1181.502483] alg: No test for adler32 (adler32-zlib) [ 1182.590528] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1182.814744] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1184.463808] Key type lgssc registered [ 1185.849899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1193.110760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1193.129578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.477447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.194912] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1204.713369] LustreError: 39981:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1204.747764] LustreError: 39981:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1206.421197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1206.838432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.297769] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1219.085642] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1225.467227] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1231.595498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1231.948443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1238.766272] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1239.185934] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1246.261709] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1252.833046] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1252.842355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.853696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1254.883187] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1254.898448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1254.903249] Lustre: Skipped 1 previous similar message [ 1254.921360] Lustre: Skipped 3 previous similar messages [ 1258.053848] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 1258.300261] Lustre: server umount lustre-MDT0000 complete [ 1260.001861] LustreError: 39982:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.026703] LustreError: 39982:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1261.047658] LustreError: 41821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1262.391704] LustreError: 39962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776519 with bad export cookie 9381507672313757891 [ 1262.397341] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1262.404787] LustreError: 39962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1263.016193] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1263.033792] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1266.232032] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1266.256570] LustreError: 39976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1267.044739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1267.053900] Lustre: Skipped 1 previous similar message [ 1268.972400] Lustre: server umount lustre-MDT0001 complete [ 1289.900518] Lustre: server umount lustre-OST0000 complete [ 1307.884230] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1311.413989] Key type lgssc unregistered [ 1311.890235] LNet: 43837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1311.902428] LNetError: 43837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1311.916790] LNet: Removed LNI 192.168.204.125@tcp [ 1313.167951] Key type .llcrypt unregistered [ 1313.169919] Key type ._llcrypt unregistered [ 1338.104088] Key type ._llcrypt registered [ 1338.106126] Key type .llcrypt registered [ 1338.276994] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1339.663553] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1339.691588] alg: No test for adler32 (adler32-zlib) [ 1340.764659] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1341.056903] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1342.723391] Key type lgssc registered [ 1343.921840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1351.767837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1351.789469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.258780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.517654] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1363.430856] LustreError: 45254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1363.460329] LustreError: 45254:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1367.883524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1368.259211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1372.372131] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1380.078719] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1386.808034] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1394.824466] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1395.228641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1398.347795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1401.777770] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1410.428373] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1423.440146] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 1423.845052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1423.849060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.353447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1424.361705] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1424.365866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1424.397421] Lustre: Skipped 2 previous similar messages [ 1427.017615] Lustre: server umount lustre-OST0000 complete [ 1445.528418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.549541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.577937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1449.579162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1451.863797] Lustre: server umount lustre-MDT0000 complete [ 1455.082249] LustreError: 45249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.112545] LustreError: 45249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1456.206017] LustreError: 45235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776713 with bad export cookie 4815656731431770599 [ 1456.222042] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1462.633692] Lustre: server umount lustre-MDT0001 complete [ 1473.536815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1476.581830] Key type lgssc unregistered [ 1476.899908] LNet: 48963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1476.909185] LNetError: 48963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1477.930276] LNet: Removed LNI 192.168.204.125@tcp [ 1478.961969] Key type .llcrypt unregistered [ 1478.964148] Key type ._llcrypt unregistered [ 1503.611997] Key type ._llcrypt registered [ 1503.613579] Key type .llcrypt registered [ 1503.753293] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1505.476857] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1505.514390] alg: No test for adler32 (adler32-zlib) [ 1506.648167] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1506.938459] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1508.631228] Key type lgssc registered [ 1509.851534] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1518.615349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1518.634302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1520.017049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1524.332433] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1529.314910] LustreError: 50377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.339348] LustreError: 50377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1533.023029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1533.539787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1538.367387] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1546.955395] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1553.741903] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1561.314508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1561.819985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1564.919986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1568.124036] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1575.566482] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1584.890713] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1589.420153] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1601.856492] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-07 6h27m36s [ 1603.686304] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1608.777195] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 1609.192656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.205885] Lustre: Skipped 1 previous similar message [ 1609.845747] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 1609.855350] Lustre: Skipped 6 previous similar messages [ 1612.617777] Lustre: server umount lustre-MDT0000 complete [ 1615.937252] LustreError: 50372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1617.144520] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1622.082860] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1622.121764] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1624.951089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1625.056910] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1625.251224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1625.326069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1626.290634] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1630.256866] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1633.385689] Lustre: DEBUG MARKER: Wait for df (36667) ... [ 1635.600444] Lustre: DEBUG MARKER: done [ 1640.929934] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1640.943825] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.958523] Lustre: Skipped 1 previous similar message [ 1640.968954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1640.970737] Lustre: Skipped 3 previous similar messages [ 1641.965220] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1646.606358] Lustre: server umount lustre-OST0000 complete [ 1652.194357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1652.195482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.205751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.205758] Lustre: Skipped 2 previous similar messages [ 1652.242214] Lustre: Skipped 1 previous similar message [ 1657.116467] Lustre: server umount lustre-MDT0000 complete [ 1659.381260] LustreError: 50377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1659.418581] LustreError: 50377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1661.639633] LustreError: 52219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788776918 with bad export cookie 11174612410274411244 [ 1661.640544] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1661.646184] LustreError: 52219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1662.087075] Lustre: server umount lustre-MDT0001 complete [ 1674.271745] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1677.275083] Key type lgssc unregistered [ 1677.556255] LNet: 55299:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1677.561714] LNetError: 55299:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1677.576660] LNet: Removed LNI 192.168.204.125@tcp [ 1678.428932] Key type .llcrypt unregistered [ 1678.439950] Key type ._llcrypt unregistered [ 1720.438462] Key type ._llcrypt registered [ 1720.440976] Key type .llcrypt registered [ 1720.584260] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1721.797439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1721.941045] alg: No test for adler32 (adler32-zlib) [ 1722.987534] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1723.218121] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1724.855521] Key type lgssc registered [ 1726.551844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1735.735470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1735.802970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.152274] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1737.184257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1737.419488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1742.302770] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1751.741803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.816916] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1751.872578] Lustre: 57522:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1752.299129] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1757.879828] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1765.781311] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1772.120115] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1779.863581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1780.184116] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1780.452563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1788.029947] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 1790.589129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1796.042810] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1802.895862] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-07 6h30m58s [ 1804.490357] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1812.975459] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1816.258940] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1816.325531] LustreError: 58775:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1836.686040] Lustre: lustre-MDT0000: Client 22f31df5-efd8-4f90-b449-0db231e81258 (at 192.168.204.25@tcp) reconnecting [ 1856.360262] LustreError: 58775:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1858.946520] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1861.745376] Lustre: DEBUG MARKER: done [ 1872.356899] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1872.368713] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.386872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1872.878268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1872.882349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1872.902591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1875.362464] Lustre: server umount lustre-OST0000 complete [ 1880.545532] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1880.557913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.575940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1883.687144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1883.689886] Lustre: Skipped 4 previous similar messages [ 1889.769612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1889.786759] Lustre: Skipped 1 previous similar message [ 1894.367282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1894.621861] Lustre: server umount lustre-MDT0000 complete [ 1898.788872] LustreError: 59382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777155 with bad export cookie 12042653519072235300 [ 1898.802581] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1898.811637] LustreError: 59382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1899.622262] Lustre: server umount lustre-MDT0001 complete [ 1910.515606] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 1913.709922] Key type lgssc unregistered [ 1914.069044] LNet: 61853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.082986] LNetError: 61853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.103341] LNet: Removed LNI 192.168.204.125@tcp [ 1915.003880] Key type .llcrypt unregistered [ 1915.005751] Key type ._llcrypt unregistered [ 1962.596847] Key type ._llcrypt registered [ 1962.599443] Key type .llcrypt registered [ 1962.731888] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 1964.397439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1964.474771] alg: No test for adler32 (adler32-zlib) [ 1965.593471] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 1966.031581] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 1967.727494] Key type lgssc registered [ 1969.022825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.910653] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1972.940517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.254664] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1974.308032] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1974.598726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1975.956215] Lustre: Failing over lustre-MDT0000 [ 1976.409182] Lustre: server umount lustre-MDT0000 complete [ 1984.111396] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 06:33:59 (1788777239) [ 1992.921098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.548402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1997.624877] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2008.678512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2008.837821] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2008.890313] Lustre: 64656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2009.249611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2014.621488] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2023.741993] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2030.496647] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2037.191059] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2037.372254] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2037.536340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2043.720467] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2044.645166] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2052.315622] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2059.425409] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2070.511862] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2070.521443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2070.549380] Lustre: Skipped 2 previous similar messages [ 2070.581320] Lustre: Skipped 2 previous similar messages [ 2075.480393] Lustre: server umount lustre-MDT0000 complete [ 2080.781457] LustreError: 65905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2080.803782] LustreError: 65905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2081.789664] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777338 with bad export cookie 7605894571086426833 [ 2081.818286] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2082.166490] Lustre: server umount lustre-MDT0001 complete [ 2086.634806] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2091.874813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2100.980268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2101.151145] Lustre: 63575:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788777342/real 1788777342] req@0000000037110a55 x1875668865428224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788777358 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2101.174791] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2102.240456] Lustre: 63575:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788777343/real 1788777343] req@00000000080e1a6d x1875668865428352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788777359 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2106.335676] Lustre: 63576:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788777347/real 1788777347] req@00000000a01accbc x1875668865428480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788777363 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2107.653487] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2107.783764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2107.894061] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2108.831771] Lustre: 63577:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788777349/real 1788777349] req@000000007c63bf3e x1875668865428736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788777365 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2110.837232] LustreError: 66505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2110.844938] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2112.492437] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2112.863335] Lustre: 63574:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788777353/real 1788777353] req@00000000ea13375d x1875668865428992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788777369 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2112.879061] Lustre: 63574:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2117.103543] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2117.126253] LustreError: 66504:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2121.118929] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2121.649396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2126.637322] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2126.829119] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2134.085178] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2140.347918] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2146.967310] Lustre: server umount lustre-MDT0000 complete [ 2147.310421] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2147.311436] LustreError: 68650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.314196] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2147.330026] Lustre: Skipped 2 previous similar messages [ 2147.359887] LustreError: 68650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2150.862372] LustreError: 64647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777407 with bad export cookie 7605894571086429108 [ 2150.867485] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2150.874681] LustreError: 64647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2151.155926] Lustre: server umount lustre-MDT0001 complete [ 2156.747842] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2166.134253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.805887] Lustre: 63574:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788777409/real 1788777409] req@000000004cee08de x1875668865454592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788777425 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2168.839734] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2175.466051] LustreError: 69428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2175.554074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2175.633633] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2181.095793] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2187.766823] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2191.651385] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.490192] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2197.539736] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2205.764231] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2211.614629] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2223.078251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2223.085886] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2223.088615] Lustre: Skipped 2 previous similar messages [ 2223.088981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2223.113896] Lustre: Skipped 2 previous similar messages [ 2224.609083] Lustre: server umount lustre-MDT0000 complete [ 2228.194187] LustreError: 71111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2228.221835] LustreError: 71111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2228.605954] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777485 with bad export cookie 7605894571086430487 [ 2228.615655] LustreError: 64648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2228.615834] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2229.017628] Lustre: server umount lustre-MDT0001 complete [ 2235.869294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2240.039210] Lustre: DEBUG MARKER: files compared the same [ 2252.949651] Lustre: server umount lustre-OST0000 complete [ 2271.271361] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2274.322601] Key type lgssc unregistered [ 2274.734770] LNet: 74385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2274.740289] LNetError: 74385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2274.767784] LNet: Removed LNI 192.168.204.125@tcp [ 2275.519322] Key type .llcrypt unregistered [ 2275.522218] Key type ._llcrypt unregistered [ 2300.771225] Key type ._llcrypt registered [ 2300.774740] Key type .llcrypt registered [ 2300.929552] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2301.788325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2301.842159] alg: No test for adler32 (adler32-zlib) [ 2302.975075] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2303.275376] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2305.079737] Key type lgssc registered [ 2306.193452] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2314.804834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2314.833800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.514975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2321.593745] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 2327.008363] LustreError: 75800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2327.035167] LustreError: 75800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2331.482550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.799427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2337.076744] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 2346.314655] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2353.641591] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2361.728201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2362.209787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2363.270872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2369.334912] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug +malloc all [ 2377.360333] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2386.913545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2386.914604] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2386.933760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2386.943529] Lustre: Skipped 1 previous similar message [ 2386.955423] Lustre: Skipped 1 previous similar message [ 2392.032178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2392.047569] Lustre: Skipped 1 previous similar message [ 2392.417325] Lustre: server umount lustre-OST0000 complete [ 2397.155233] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2397.163690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2397.181426] Lustre: Skipped 1 previous similar message [ 2397.193991] Lustre: Skipped 1 previous similar message [ 2402.271846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2402.279047] Lustre: Skipped 1 previous similar message [ 2403.290385] Lustre: server umount lustre-MDT0000 complete [ 2407.397213] LustreError: 77097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2407.425419] LustreError: 77097:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2408.028778] LustreError: 78581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777665 with bad export cookie 16425476828560760376 [ 2408.030650] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2408.041117] LustreError: 78581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2408.590859] Lustre: server umount lustre-MDT0001 complete [ 2420.850141] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2423.774551] Key type lgssc unregistered [ 2424.055206] LNet: 79332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2424.067187] LNetError: 79332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2424.082424] LNet: Removed LNI 192.168.204.125@tcp [ 2425.107157] Key type .llcrypt unregistered [ 2425.112730] Key type ._llcrypt unregistered [ 2452.158643] Key type ._llcrypt registered [ 2452.161258] Key type .llcrypt registered [ 2452.294499] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2453.839413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2453.996256] alg: No test for adler32 (adler32-zlib) [ 2455.129993] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2455.460022] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2457.162815] Key type lgssc registered [ 2458.557518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2466.688150] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2466.711379] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2482.336725] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2487.455328] LustreError: 80717:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.125@tcp: failed processing log, type 4: rc = -110 [ 2514.143279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.852474] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2530.331412] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2539.556173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2539.733630] Lustre: *** cfs_fail_loc=706, val=0*** [ 2540.263408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2540.350723] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2545.396328] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2545.645346] LustreError: 80742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2545.669807] LustreError: 80742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2548.723519] LustreError: 80743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2548.752238] LustreError: 80743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2553.842341] LustreError: 80742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.876779] LustreError: 80742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2555.669996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2556.074655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2560.921783] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2570.067680] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2577.947435] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2581.474346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2581.475785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2581.492819] LustreError: Skipped 1 previous similar message [ 2581.497000] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2581.503932] Lustre: Skipped 1 previous similar message [ 2581.513194] Lustre: Skipped 1 previous similar message [ 2586.646836] Lustre: server umount lustre-OST0000 complete [ 2591.714538] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2591.722110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2591.739981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2597.136672] Lustre: server umount lustre-MDT0000 complete [ 2597.344974] LustreError: 81804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2597.362674] LustreError: 81804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2600.884440] LustreError: 80723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788777857 with bad export cookie 6507115501892794164 [ 2600.898932] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2601.298077] Lustre: server umount lustre-MDT0001 complete [ 2612.926865] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 2616.340898] Key type lgssc unregistered [ 2616.662271] LNet: 84337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2616.678640] LNetError: 84337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2616.698872] LNet: Removed LNI 192.168.204.125@tcp [ 2617.713898] Key type .llcrypt unregistered [ 2617.716100] Key type ._llcrypt unregistered [ 2645.585470] Key type ._llcrypt registered [ 2645.592025] Key type .llcrypt registered [ 2645.728327] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2646.622196] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2646.651823] alg: No test for adler32 (adler32-zlib) [ 2647.767067] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 2648.111233] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 2649.863173] Key type lgssc registered [ 2651.262106] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2659.887029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2659.899603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.193735] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2673.649706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2674.605607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2679.784421] LustreError: 86440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.815081] LustreError: 86440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2680.246549] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2684.901841] LustreError: 86444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2684.938412] LustreError: 86444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2690.017982] LustreError: 86440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2690.038141] LustreError: 86440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2692.397787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2692.816272] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2692.843983] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2692.866235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2698.210084] LustreError: 86440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2700.165324] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2709.487784] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.515830] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 2711.649412] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2711.657933] Lustre: MGS: Client 12c94733-6840-49ed-9978-4b6391da4695 (at 0@lo) reconnecting [ 2711.674060] LustreError: 86454:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2711.976472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2712.082863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2717.486826] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2732.519938] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.520613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.532923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.548294] Lustre: Skipped 1 previous similar message [ 2737.644620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.657190] Lustre: Skipped 1 previous similar message [ 2738.348696] Lustre: server umount lustre-OST0000 complete [ 2742.754063] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.759625] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2742.760253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2742.760259] Lustre: Skipped 1 previous similar message [ 2742.785925] Lustre: Skipped 1 previous similar message [ 2747.878331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2747.889815] Lustre: Skipped 2 previous similar messages [ 2748.650112] Lustre: server umount lustre-MDT0000 complete [ 2752.996277] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2753.019519] LustreError: 86443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2754.488796] Lustre: server umount lustre-MDT0001 complete [ 2759.476778] Lustre: server umount MGS complete [ 2773.604903] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 06:47:09 (1788778029) [ 2809.670471] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 2822.677184] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 2862.330898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2866.773066] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2876.705419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2876.918723] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2876.978755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2877.052151] Lustre: lustre-MDT0001: new disk, initializing [ 2877.148084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2877.180052] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2877.188075] Lustre: 93631:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2877.196340] LustreError: 93631:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2881.534737] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2891.760072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2892.044526] Lustre: lustre-OST0000: new disk, initializing [ 2892.049662] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2892.056281] Lustre: 94400:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2892.130726] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.042719] Lustre: 94688:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000e324d73b x1875669580555520/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788778172 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2899.121924] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2909.351336] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2917.096688] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2917.118950] Lustre: MGS: Client 2e7fd7d8-32fb-402e-9f64-ee4e0941f430 (at 0@lo) reconnecting [ 2917.162927] Lustre: 95445:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2917.181527] LustreError: 93617:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2917.197538] LustreError: 93617:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2917.232106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2917.307333] Lustre: lustre-MDT0000: new disk, initializing [ 2917.436543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2917.469109] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2917.496037] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2918.568182] Lustre: 95470:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ced60d1f x1875669580562432/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788778191 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2923.031632] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 2923.805985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2923.813945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2923.869615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2927.660969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2927.674575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2949.611945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2949.614484] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2949.652945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2949.679427] Lustre: Skipped 1 previous similar message [ 2955.217152] Lustre: server umount lustre-OST0000 complete [ 2964.962673] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2964.963201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2964.964779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2964.964785] Lustre: Skipped 2 previous similar messages [ 2965.009982] Lustre: Skipped 2 previous similar messages [ 2966.468292] Lustre: server umount lustre-MDT0000 complete [ 2970.082212] LustreError: 93601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2970.161865] LustreError: 93601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2976.965354] Lustre: server umount lustre-MDT0001 complete [ 2980.679296] Lustre: server umount MGS complete [ 2993.874736] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 06:50:49 (1788778249) [ 3013.722959] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3016.561267] Key type lgssc unregistered [ 3016.851328] LNet: 97970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3016.861462] LNetError: 97970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3016.876713] LNet: Removed LNI 192.168.204.125@tcp [ 3017.682718] Key type .llcrypt unregistered [ 3017.685359] Key type ._llcrypt unregistered [ 3035.109082] Key type ._llcrypt registered [ 3035.114331] Key type .llcrypt registered [ 3035.222408] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3036.366046] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3036.419651] alg: No test for adler32 (adler32-zlib) [ 3037.472654] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3037.712066] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3039.359190] Key type lgssc registered [ 3040.398714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3047.436812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3047.461923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3048.866037] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3052.993815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3058.150912] LustreError: 99191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3058.178101] LustreError: 99191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3062.324932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3063.024637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3069.970693] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3079.974510] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3086.584747] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3095.350486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3095.643351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3102.053402] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3110.455203] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3118.197340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3128.804280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3128.808527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3128.814495] Lustre: Skipped 1 previous similar message [ 3128.838627] Lustre: Skipped 1 previous similar message [ 3133.923248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3133.930392] Lustre: Skipped 1 previous similar message [ 3134.899825] Lustre: server umount lustre-OST0000 complete [ 3139.045625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3139.078800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3139.087279] Lustre: Skipped 1 previous similar message [ 3144.160251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3144.174408] Lustre: Skipped 1 previous similar message [ 3144.851682] Lustre: server umount lustre-MDT0000 complete [ 3149.282092] LustreError: 100298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3149.302974] LustreError: 100298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3149.356899] LustreError: 99171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778406 with bad export cookie 11946329073819419025 [ 3149.359676] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3149.375781] LustreError: 99171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3149.898541] Lustre: server umount lustre-MDT0001 complete [ 3161.507244] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3164.321918] Key type lgssc unregistered [ 3164.655825] LNet: 102748:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3164.659619] LNetError: 102748:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3164.681200] LNet: Removed LNI 192.168.204.125@tcp [ 3165.629209] Key type .llcrypt unregistered [ 3165.632774] Key type ._llcrypt unregistered [ 3182.090696] Key type ._llcrypt registered [ 3182.091947] Key type .llcrypt registered [ 3182.227350] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3182.989026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3183.020121] alg: No test for adler32 (adler32-zlib) [ 3184.309362] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3184.654668] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3186.488339] Key type lgssc registered [ 3188.103825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3195.839877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3195.846066] LustreError: 103827:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3195.871543] LustreError: 103828:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3195.881120] LustreError: 103827:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4974 [ 3195.891938] LustreError: 103827:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3195.893529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.904395] LustreError: 103827:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3197.360754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3205.154435] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3205.390659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3209.117285] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3214.513868] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lsmod [ 3217.915930] LustreError: 105034:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3217.923549] LustreError: 105033:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3217.931849] LustreError: 105034:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3217.938558] LustreError: 105034:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3217.947969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3217.954635] LustreError: 105034:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3218.455764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3223.532438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3223.535828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3223.561363] Lustre: Skipped 1 previous similar message [ 3223.579877] Lustre: Skipped 2 previous similar messages [ 3227.916568] Lustre: server umount lustre-MDT0000 complete [ 3228.643098] LustreError: 103865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3228.654513] LustreError: 103865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3230.712346] LustreError: 103866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3230.738912] LustreError: 103866:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3231.496172] LustreError: 103843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778488 with bad export cookie 5275064179616646613 [ 3231.508972] LustreError: 103843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3231.509884] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3231.866452] Lustre: server umount lustre-MDT0001 complete [ 3245.320464] Lustre: server umount lustre-OST0000 complete [ 3253.784435] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.191069] LustreError: 106098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3254.265068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3259.056718] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3259.367892] LustreError: 106099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3264.499064] LustreError: 106098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.208685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3269.497728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.905399] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3284.246399] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3289.957082] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3297.456871] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3297.698305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3299.766311] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3303.103725] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3311.497325] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3324.407593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3324.408661] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3324.423378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3324.454342] Lustre: Skipped 2 previous similar messages [ 3327.511916] Lustre: server umount lustre-OST0000 complete [ 3334.625743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3334.632584] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3334.644159] Lustre: Skipped 1 previous similar message [ 3334.659911] Lustre: Skipped 2 previous similar messages [ 3339.744882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3339.751383] Lustre: Skipped 1 previous similar message [ 3344.865550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3344.873870] Lustre: Skipped 1 previous similar message [ 3346.399137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3346.579697] Lustre: server umount lustre-MDT0000 complete [ 3349.986435] LustreError: 106095:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.997646] LustreError: 106095:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3350.349321] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778607 with bad export cookie 5275064179616647299 [ 3350.355495] LustreError: 106079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3350.356859] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3350.803255] Lustre: server umount lustre-MDT0001 complete [ 3362.931021] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3365.743729] Key type lgssc unregistered [ 3366.012356] LNet: 109631:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3366.033219] LNetError: 109631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3366.053768] LNet: Removed LNI 192.168.204.125@tcp [ 3366.779203] Key type .llcrypt unregistered [ 3366.781580] Key type ._llcrypt unregistered [ 3389.509400] Key type ._llcrypt registered [ 3389.514240] Key type .llcrypt registered [ 3389.599457] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3390.439910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3390.482039] alg: No test for adler32 (adler32-zlib) [ 3391.528379] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3391.806646] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3393.560510] Key type lgssc registered [ 3394.820782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3403.299854] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3403.339562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3405.123865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3409.753740] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3415.524084] LustreError: 111042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3415.565839] LustreError: 111042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3420.642633] LustreError: 111043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.112811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.668440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3427.144481] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3435.361504] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3441.959726] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3449.998623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3450.354606] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 3450.427712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.494467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3456.572404] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3465.205703] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3474.584159] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3480.400742] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3486.177343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3486.188785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3486.203863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3488.234153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3488.234166] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3488.255767] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3492.373951] Lustre: server umount lustre-OST0000 complete [ 3498.478256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.499533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3498.510111] Lustre: Skipped 2 previous similar messages [ 3503.587540] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.592152] Lustre: Skipped 1 previous similar message [ 3504.041549] Lustre: server umount lustre-MDT0000 complete [ 3508.549091] LustreError: 111025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778765 with bad export cookie 16010138203079409514 [ 3508.553383] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3508.563835] LustreError: 111025:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3508.916415] Lustre: server umount lustre-MDT0001 complete [ 3521.256938] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3525.039192] Key type lgssc unregistered [ 3525.435650] LNet: 114681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3525.439954] LNetError: 114681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3526.504505] LNet: Removed LNI 192.168.204.125@tcp [ 3527.631472] Key type .llcrypt unregistered [ 3527.636917] Key type ._llcrypt unregistered [ 3548.232539] Key type ._llcrypt registered [ 3548.235672] Key type .llcrypt registered [ 3548.373737] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3549.800846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3549.859103] alg: No test for adler32 (adler32-zlib) [ 3551.141931] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3551.465622] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3553.175459] Key type lgssc registered [ 3554.701094] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3564.284537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3564.306627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3565.897235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3570.450420] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3575.270029] LustreError: 115903:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3575.290552] LustreError: 115903:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3580.392051] LustreError: 115904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3580.966231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3581.386323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3586.642160] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3596.437625] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3603.380835] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3611.348175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3611.732490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3618.420392] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3626.151466] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3635.341796] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3646.946248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3646.966909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3646.989492] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.046636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3651.061545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3651.070584] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3652.523419] Lustre: server umount lustre-OST0000 complete [ 3658.208546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3658.221355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3658.255581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3658.271861] Lustre: Skipped 1 previous similar message [ 3661.280788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3661.282677] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3663.881404] Lustre: server umount lustre-MDT0000 complete [ 3666.407804] LustreError: 115904:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3666.436094] LustreError: 115904:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3668.117690] LustreError: 115883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788778925 with bad export cookie 3378357641635841953 [ 3668.124557] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3668.126897] LustreError: 115883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3668.703354] Lustre: server umount lustre-MDT0001 complete [ 3681.881418] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3685.192276] Key type lgssc unregistered [ 3685.510693] LNet: 119471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3685.525086] LNetError: 119471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3685.543945] LNet: Removed LNI 192.168.204.125@tcp [ 3686.561367] Key type .llcrypt unregistered [ 3686.565086] Key type ._llcrypt unregistered [ 3712.523387] Key type ._llcrypt registered [ 3712.525933] Key type .llcrypt registered [ 3712.705131] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3714.252325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3714.286829] alg: No test for adler32 (adler32-zlib) [ 3715.338552] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3715.566371] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3717.231178] Key type lgssc registered [ 3718.423709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3725.634320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3725.646983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3726.931185] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3730.504238] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3736.036600] LustreError: 120886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3736.058278] LustreError: 120886:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3739.353811] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3739.858484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3744.784987] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3752.447599] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3758.345125] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3765.223714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3765.506279] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3771.103528] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3771.718223] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3777.943696] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3788.250486] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3791.987601] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3791.992419] Lustre: Skipped 2 previous similar messages [ 3796.113599] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3796.121241] Lustre: Skipped 2 previous similar messages [ 3798.601810] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3798.650978] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3802.670989] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.25@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3802.677582] Lustre: Skipped 2 previous similar messages [ 3804.927857] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.25@tcp 192.168.0.[2,10]@tcp [ 3816.957194] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.25/24@tcp in log lustre-MDT0000 [ 3816.968486] Lustre: Skipped 2 previous similar messages [ 3826.264126] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.25/24@tcp [ 3826.280271] Lustre: Skipped 1 previous similar message [ 3836.389276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3836.392892] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3836.421519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3836.433365] Lustre: Skipped 1 previous similar message [ 3840.528186] Lustre: server umount lustre-OST0000 complete [ 3846.624957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.648210] Lustre: Skipped 1 previous similar message [ 3846.651676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3846.662719] Lustre: Skipped 1 previous similar message [ 3851.511497] Lustre: server umount lustre-MDT0000 complete [ 3851.745848] LustreError: 121617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.769476] LustreError: 121617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3856.266282] LustreError: 123515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779113 with bad export cookie 8307898945108967031 [ 3856.268516] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3856.280573] LustreError: 123515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3856.668645] Lustre: server umount lustre-MDT0001 complete [ 3868.231761] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 3871.433379] Key type lgssc unregistered [ 3871.820097] LNet: 125773:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3871.847691] LNetError: 125773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3871.889629] LNet: Removed LNI 192.168.204.125@tcp [ 3873.276811] Key type .llcrypt unregistered [ 3873.280614] Key type ._llcrypt unregistered [ 3911.323361] Key type ._llcrypt registered [ 3911.325854] Key type .llcrypt registered [ 3911.501622] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 3913.322510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3913.400354] alg: No test for adler32 (adler32-zlib) [ 3914.626280] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 3914.965672] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 3916.647168] Key type lgssc registered [ 3918.103166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3926.942092] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3926.963952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3928.639714] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3928.683639] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.25/24@tcp [ 3928.686484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3934.080594] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3938.786947] LustreError: 127353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3938.819415] LustreError: 127353:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3943.910207] LustreError: 127352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3944.296506] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3944.816878] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3944.844681] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.25/24@tcp [ 3944.921434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3950.515593] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3960.042271] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3967.773853] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3975.003426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3975.591769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.074662] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 3990.339674] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3998.141786] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4013.541994] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4013.559177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4013.567538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4013.580896] Lustre: Skipped 1 previous similar message [ 4013.595425] Lustre: Skipped 1 previous similar message [ 4017.686274] Lustre: server umount lustre-OST0000 complete [ 4021.727926] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4021.735113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4021.752564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.777279] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4023.778355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4023.796836] Lustre: Skipped 1 previous similar message [ 4028.028683] Lustre: server umount lustre-MDT0000 complete [ 4028.906830] LustreError: 128305:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.928494] LustreError: 128305:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4032.197850] LustreError: 127334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779289 with bad export cookie 16184706223139003784 [ 4032.201113] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4032.216867] LustreError: 127334:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4032.576679] Lustre: server umount lustre-MDT0001 complete [ 4045.579851] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4048.724682] Key type lgssc unregistered [ 4049.045583] LNet: 130963:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4049.059144] LNetError: 130963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4049.094141] LNet: Removed LNI 192.168.204.125@tcp [ 4050.181234] Key type .llcrypt unregistered [ 4050.183644] Key type ._llcrypt unregistered [ 4080.284914] Key type ._llcrypt registered [ 4080.290571] Key type .llcrypt registered [ 4080.433363] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 4095.580580] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4096.529381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4096.574451] alg: No test for adler32 (adler32-zlib) [ 4097.650969] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 4097.834757] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4099.479196] Key type lgssc registered [ 4100.682919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4135.007306] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4135.033449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4136.307504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4136.334939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4136.413509] Lustre: lustre-MDT0000: new disk, initializing [ 4136.501553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4136.532554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4141.715078] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4153.095371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.256178] Lustre: 134360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4153.316641] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4153.321246] Lustre: Skipped 1 previous similar message [ 4153.416945] Lustre: lustre-MDT0001: new disk, initializing [ 4153.550784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4153.604544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4153.634026] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4158.073736] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4168.603168] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4175.357564] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4182.749733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.091342] Lustre: lustre-OST0000: new disk, initializing [ 4183.096094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4183.105399] Lustre: 136329:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4183.211919] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4189.774300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4189.798585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4189.968038] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4190.006626] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4201.340563] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4207.036185] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4207.269380] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4212.834869] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4213.114468] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4215.777394] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4215.785181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4215.802810] Lustre: Skipped 1 previous similar message [ 4220.906028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4220.918859] Lustre: Skipped 2 previous similar messages [ 4221.397874] Lustre: server umount lustre-OST0000 complete [ 4225.505330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4225.512985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4225.538325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4231.137948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4231.139934] Lustre: Skipped 2 previous similar messages [ 4231.612838] Lustre: server umount lustre-MDT0000 complete [ 4235.626981] LustreError: 134352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779492 with bad export cookie 11871945889495134243 [ 4235.633316] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4235.646357] LustreError: 134352:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4236.026107] Lustre: server umount lustre-MDT0001 complete [ 4244.435799] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4244.998182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4249.755939] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4250.082461] LustreError: 138547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4250.091728] LustreError: 138547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4254.183662] LustreError: 138546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4258.391534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.743041] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4264.773295] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4273.060136] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4278.804308] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4285.559047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4286.040237] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4293.959350] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4302.511790] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4308.083530] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4308.348930] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4313.598752] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4313.756698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4322.135362] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4322.347299] Lustre: lustre-OST0001: new disk, initializing [ 4322.353696] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4322.364094] Lustre: 141919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4322.517980] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4329.567959] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4332.142916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4332.157824] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4332.243780] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4338.622269] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4338.894674] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4343.672369] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4343.884908] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4351.846770] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b55206cd800.ost_server_uuid 50 [ 4354.323589] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b55206cd800.ost_server_uuid in IDLE state after 0 sec [ 4368.355604] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4368.366378] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4368.372531] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4368.391957] Lustre: Skipped 2 previous similar messages [ 4368.422172] Lustre: Skipped 1 previous similar message [ 4370.412775] Lustre: server umount lustre-OST0001 complete [ 4373.477693] LustreError: 140368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4373.501410] LustreError: 140368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4375.523288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4375.533802] LustreError: Skipped 1 previous similar message [ 4375.546525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4376.548417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4376.560531] Lustre: Skipped 1 previous similar message [ 4378.598816] LustreError: 140369:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4378.617544] LustreError: 140369:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4380.546857] Lustre: server umount lustre-OST0000 complete [ 4386.786105] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4386.797308] LustreError: Skipped 1 previous similar message [ 4386.800736] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4386.817251] Lustre: Skipped 1 previous similar message [ 4391.270896] Lustre: server umount lustre-MDT0000 complete [ 4393.956309] LustreError: 138547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4393.976766] LustreError: 138547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4395.078262] LustreError: 138528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779652 with bad export cookie 11871945889495135125 [ 4395.088794] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4395.093833] LustreError: 138528:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4395.525400] Lustre: server umount lustre-MDT0001 complete [ 4415.429896] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4418.219468] Key type lgssc unregistered [ 4418.589724] LNet: 144923:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4418.597687] LNetError: 144923:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4418.617882] LNet: Removed LNI 192.168.204.125@tcp [ 4419.651925] Key type .llcrypt unregistered [ 4419.658050] Key type ._llcrypt unregistered [ 4459.427617] Key type ._llcrypt registered [ 4459.431983] Key type .llcrypt registered [ 4459.532907] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 4472.076218] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4472.797474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4472.863172] alg: No test for adler32 (adler32-zlib) [ 4473.982870] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 4474.284540] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4475.959203] Key type lgssc registered [ 4477.072914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4509.741540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4509.787281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.038347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4511.074714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4511.182374] Lustre: lustre-MDT0000: new disk, initializing [ 4511.259201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4511.276974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4514.843703] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4523.802219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.907928] Lustre: 149051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4523.957891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4523.963698] Lustre: Skipped 1 previous similar message [ 4524.024758] Lustre: lustre-MDT0001: new disk, initializing [ 4524.093581] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4524.124569] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4524.139166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4528.019557] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4536.597647] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4542.211326] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4548.881015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4549.185118] Lustre: lustre-OST0000: new disk, initializing [ 4549.189667] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4549.194106] Lustre: 151021:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4549.267909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4555.133864] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4558.906189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4558.915785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4559.004037] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4563.187944] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4567.205372] Lustre: Failing over lustre-OST0000 [ 4567.323832] Lustre: server umount lustre-OST0000 complete [ 4585.403740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4585.680516] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4585.703877] Lustre: Skipped 1 previous similar message [ 4585.736578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4585.767428] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4586.719616] Lustre: 147276:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788779826/real 1788779826] req@00000000d68d9caf x1875671495245312/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788779842 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4590.646515] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4590.815174] Lustre: 147274:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788779831/real 1788779831] req@00000000b34bd36e x1875671495246208/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788779847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4590.847281] Lustre: 147274:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4591.034195] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4591.117874] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4591.119377] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4592.160352] Lustre: 147277:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788779831/real 1788779831] req@000000001504f133 x1875671495246336/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788779847 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4593.930648] Lustre: Failing over lustre-MDT0000 [ 4594.142360] Lustre: server umount lustre-MDT0000 complete [ 4595.698451] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4595.738282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4595.800566] LustreError: 150144:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4595.871143] LustreError: 150144:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4595.984335] Lustre: 147275:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788779836/real 1788779836] req@00000000a561f4dd x1875671495247232/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788779852 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4600.287295] Lustre: 147277:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788779841/real 1788779841] req@0000000005c65361 x1875671495248512/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788779857 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4600.886300] LustreError: 149058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4600.914619] LustreError: 149058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4606.014214] LustreError: 149059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4606.026612] LustreError: 149059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4610.015122] Lustre: 149055:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788779851/real 1788779851] req@00000000bd136f96 x1875671495255936/t0(0) o101->MGC192.168.204.125@tcp@0@lo:26/25 lens 328/344 e 0 to 1 dl 1788779867 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4610.046985] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4610.085202] Lustre: 149055:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.125@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 4610.089353] LustreError: 149062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4610.133069] LustreError: 149062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4614.210849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4615.146290] LustreError: 152934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4615.166229] LustreError: 152934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4620.687719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.761372] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4622.389256] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4624.898446] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4625.915903] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4625.922884] Lustre: Skipped 1 previous similar message [ 4625.937081] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4625.967267] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4636.131662] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4636.133312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4636.157346] Lustre: Skipped 2 previous similar messages [ 4636.167612] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4636.172157] Lustre: Skipped 1 previous similar message [ 4640.169703] Lustre: server umount lustre-OST0000 complete [ 4646.369873] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.377967] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4646.395086] Lustre: Skipped 1 previous similar message [ 4646.408995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.423681] Lustre: Skipped 1 previous similar message [ 4650.471619] Lustre: server umount lustre-MDT0000 complete [ 4651.488787] LustreError: 149057:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4651.520880] LustreError: 149057:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4654.544113] LustreError: 149044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788779911 with bad export cookie 10371859171097321585 [ 4654.548569] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4654.551939] LustreError: 149044:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4654.891672] Lustre: server umount lustre-MDT0001 complete [ 4668.727319] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 4672.325742] Key type lgssc unregistered [ 4672.698097] LNet: 154676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4672.716038] LNetError: 154676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4672.751293] LNet: Removed LNI 192.168.204.125@tcp [ 4673.873451] Key type .llcrypt unregistered [ 4673.878989] Key type ._llcrypt unregistered [ 4713.259628] Key type ._llcrypt registered [ 4713.261590] Key type .llcrypt registered [ 4713.343755] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4714.298416] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4714.358757] alg: No test for adler32 (adler32-zlib) [ 4715.475706] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 4715.797585] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 4717.481740] Key type lgssc registered [ 4718.586599] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4726.645756] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4726.663836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4728.144965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4733.278238] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4738.530906] LustreError: 156193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4738.556413] LustreError: 156193:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4742.838504] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4743.650188] LustreError: 156192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4747.146154] LustreError: 156221:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4747.155143] LustreError: 156221:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4747.163957] LustreError: 156221:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4747.561706] Lustre: server umount lustre-MDT0000 complete [ 4763.656202] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 4775.274064] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 4816.298956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4816.701777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4816.747310] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4816.876716] Lustre: lustre-MDT0000: new disk, initializing [ 4817.013881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4817.039197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4821.660188] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4832.184848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4832.321780] Lustre: 159528:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4832.377497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4832.380725] Lustre: Skipped 1 previous similar message [ 4832.440180] Lustre: lustre-MDT0001: new disk, initializing [ 4832.537276] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4832.576177] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4832.584315] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4837.671933] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4847.250973] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4853.308483] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4859.559502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4859.817586] Lustre: lustre-OST0000: new disk, initializing [ 4859.822643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4859.830105] Lustre: 161498:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4859.932099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4865.905380] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 4868.728120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4868.746069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4868.929244] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4875.409486] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4880.126408] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4881.648953] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 07:22:17 (1788780137) [ 5134.825226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5134.836304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5134.848691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5135.345370] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5135.382393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5135.385472] Lustre: Skipped 2 previous similar messages [ 5138.685909] Lustre: server umount lustre-MDT0000 complete [ 5140.448883] LustreError: 162218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5140.464990] LustreError: 162218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5142.839830] LustreError: 159519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780399 with bad export cookie 3052616149344061255 [ 5142.841762] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5142.846921] LustreError: 159519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5145.571228] LustreError: 159536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5145.572416] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5145.601673] Lustre: Skipped 1 previous similar message [ 5145.610319] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5147.616950] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5152.742459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5152.742459] LustreError: 160947:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5152.742474] LustreError: 160947:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5157.343178] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5157.524080] Lustre: server umount lustre-MDT0001 complete [ 5176.287235] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5176.495424] Lustre: server umount lustre-OST0000 complete [ 5187.373180] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5199.412433] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5239.926840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5240.202841] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5240.270786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5240.381487] Lustre: lustre-MDT0000: new disk, initializing [ 5240.492098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.511939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5245.914796] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5257.191106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.301652] Lustre: 166163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5257.355411] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5257.358947] Lustre: Skipped 1 previous similar message [ 5257.456342] Lustre: lustre-MDT0001: new disk, initializing [ 5257.561747] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5257.612511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5257.624938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5263.308612] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5274.089075] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5280.013486] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5286.435729] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5286.646979] Lustre: lustre-OST0000: new disk, initializing [ 5286.649835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5286.663265] Lustre: 168135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5286.745611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5293.183533] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5294.709830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5294.720988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5294.874815] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5303.127462] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5307.938985] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5308.190756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5313.006429] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5313.252935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5315.566031] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5315.566870] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5315.583894] LustreError: Skipped 1 previous similar message [ 5315.598155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5329.375294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5329.620024] Lustre: server umount lustre-OST0000 complete [ 5334.496801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5334.503592] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5334.514222] Lustre: Skipped 1 previous similar message [ 5334.527911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5334.535498] Lustre: Skipped 5 previous similar messages [ 5340.686517] Lustre: server umount lustre-MDT0000 complete [ 5341.153984] LustreError: 166175:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5341.176816] LustreError: 166175:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5344.695657] LustreError: 166157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780601 with bad export cookie 3052616149344220085 [ 5344.699447] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5344.714549] LustreError: 166157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5345.006201] Lustre: server umount lustre-MDT0001 complete [ 5354.967671] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:30:10 (1788780610) [ 5373.139855] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5384.487446] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5420.959609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5421.159938] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5421.189732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5421.262617] Lustre: lustre-MDT0000: new disk, initializing [ 5421.322457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5421.341631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5425.133182] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5436.271631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5436.519254] Lustre: 172900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5436.615680] LustreError: 172912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5436.900612] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5442.408327] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5453.453519] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5460.079583] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5468.722095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5469.011410] Lustre: lustre-OST0000: new disk, initializing [ 5469.013790] Lustre: Skipped 1 previous similar message [ 5469.018533] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5469.024506] Lustre: Skipped 2 previous similar messages [ 5469.030194] Lustre: 174881:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5476.208408] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5479.020889] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5479.038292] Lustre: Skipped 1 previous similar message [ 5479.043144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5479.160482] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5487.588767] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5499.892383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5499.895139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5499.912761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5499.925347] Lustre: Skipped 2 previous similar messages [ 5499.959339] Lustre: Skipped 3 previous similar messages [ 5501.439917] Lustre: server umount lustre-OST0000 complete [ 5508.580482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5508.593779] LustreError: Skipped 1 previous similar message [ 5512.516536] Lustre: server umount lustre-MDT0000 complete [ 5515.232603] LustreError: 173935:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5515.251950] LustreError: 173935:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5516.965835] LustreError: 172894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780773 with bad export cookie 3052616149344220967 [ 5516.970943] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5516.983845] LustreError: 172894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5517.603090] Lustre: server umount lustre-MDT0001 complete [ 5529.090933] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 07:33:04 (1788780784) [ 5548.562132] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 5560.070987] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5599.724115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5599.965336] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5599.992162] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5600.140694] Lustre: lustre-MDT0000: new disk, initializing [ 5600.321077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5600.334512] Lustre: Skipped 2 previous similar messages [ 5600.382597] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5605.179155] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5616.097964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5616.306080] Lustre: 179209:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5616.671194] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5622.190869] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5632.877469] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5640.688683] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5648.091193] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5648.429225] Lustre: 181183:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5656.243628] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5658.669215] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5658.732538] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5666.757848] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5679.075574] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5679.077452] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5679.085693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5679.085701] Lustre: Skipped 3 previous similar messages [ 5679.094299] LustreError: Skipped 1 previous similar message [ 5679.126741] Lustre: Skipped 3 previous similar messages [ 5684.216593] LustreError: 181165:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5684.499441] Lustre: server umount lustre-OST0000 complete [ 5695.421403] Lustre: server umount lustre-MDT0000 complete [ 5699.746170] LustreError: 179199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788780956 with bad export cookie 3052616149344222052 [ 5699.749077] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5699.755417] LustreError: 179199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5700.135741] Lustre: server umount lustre-MDT0001 complete [ 5714.480433] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5717.738445] Key type lgssc unregistered [ 5718.155695] LNet: 183041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5718.162386] LNetError: 183041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5718.181408] LNet: Removed LNI 192.168.204.125@tcp [ 5719.279294] Key type .llcrypt unregistered [ 5719.281518] Key type ._llcrypt unregistered [ 5752.090591] Key type ._llcrypt registered [ 5752.095310] Key type .llcrypt registered [ 5752.179095] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5753.302458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5753.326125] alg: No test for adler32 (adler32-zlib) [ 5754.456829] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 5754.752260] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5756.535687] Key type lgssc registered [ 5757.680891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5766.254414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5766.301192] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5767.801439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5772.732878] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5777.376293] LustreError: 184461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5777.419958] LustreError: 184461:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5782.499618] LustreError: 184462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.159956] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5783.445016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5788.804298] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5797.550252] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5805.349582] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5812.889717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5813.248728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5820.423841] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5829.187456] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5837.809384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5858.299463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5858.311782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5858.340655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5858.355973] Lustre: Skipped 1 previous similar message [ 5863.400381] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5863.415755] Lustre: Skipped 1 previous similar message [ 5864.138263] Lustre: server umount lustre-OST0000 complete [ 5870.562224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5870.577288] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5870.593423] Lustre: Skipped 1 previous similar message [ 5870.601291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5873.634293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5873.636200] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5878.758370] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5878.771053] Lustre: Skipped 2 previous similar messages [ 5883.359388] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5883.531632] Lustre: server umount lustre-MDT0000 complete [ 5883.873310] LustreError: 185653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5883.885944] LustreError: 185653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5886.852545] LustreError: 184442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781143 with bad export cookie 14437451611499092044 [ 5886.855548] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5886.878340] LustreError: 184442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5887.207294] Lustre: server umount lustre-MDT0001 complete [ 5902.547272] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 5905.506808] Key type lgssc unregistered [ 5905.809755] LNet: 188026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5905.818862] LNetError: 188026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5905.858123] LNet: Removed LNI 192.168.204.125@tcp [ 5907.007215] Key type .llcrypt unregistered [ 5907.012553] Key type ._llcrypt unregistered [ 5936.106169] Key type ._llcrypt registered [ 5936.108295] Key type .llcrypt registered [ 5936.243085] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 5937.281211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5937.445983] alg: No test for adler32 (adler32-zlib) [ 5938.501629] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 5938.742990] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 5940.391188] Key type lgssc registered [ 5941.509400] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5949.868218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5949.879351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5951.291292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5955.524453] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5961.710649] LustreError: 189440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5961.731563] LustreError: 189440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5966.004925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5966.573979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5971.574049] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 5980.452321] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5987.192290] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5995.431328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5996.043309] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5997.115485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6003.004240] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6011.364111] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6019.671593] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 6021.608161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6021.610640] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6021.635644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6021.650742] Lustre: Skipped 1 previous similar message [ 6024.765667] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 6029.908306] Lustre: lustre-OST0000: Not available for connect from 192.168.204.25@tcp (stopping) [ 6029.925067] Lustre: Skipped 2 previous similar messages [ 6033.376531] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6033.752230] Lustre: server umount lustre-OST0000 complete [ 6042.204526] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9b55109bd800.ost_server_uuid 50 [ 6044.826399] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6058.465654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6058.471505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6058.479254] Lustre: Skipped 1 previous similar message [ 6058.482578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6058.484736] Lustre: Skipped 2 previous similar messages [ 6061.540460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6063.980395] Lustre: server umount lustre-MDT0000 complete [ 6066.655990] LustreError: 190167:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6066.669658] LustreError: 190167:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6068.814883] LustreError: 189419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781325 with bad export cookie 18153413276178858723 [ 6068.822082] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6068.838028] LustreError: 189419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6069.315123] Lustre: server umount lustre-MDT0001 complete [ 6080.312772] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 07:42:15 (1788781335) [ 6087.572074] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.158210] LustreError: 193356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6088.288443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6093.290588] LustreError: 193357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.586813] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6098.404037] LustreError: 193356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6103.325945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6103.524165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6103.536867] Lustre: Skipped 2 previous similar messages [ 6103.751230] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6108.622367] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6117.720781] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6124.381890] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6132.050292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6132.430718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6133.511903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6139.273471] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6147.369391] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6154.354054] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6154.620739] Lustre: lustre-OST0001: new disk, initializing [ 6154.624234] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6154.631856] Lustre: 196197:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6154.761511] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6162.139893] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6163.082435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6163.098254] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6163.267230] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6172.911952] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6178.787680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6178.816818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6182.811805] Lustre: server umount lustre-OST0000 complete [ 6183.940848] LustreError: 195179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6183.972630] LustreError: 195179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6189.138812] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6189.334347] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6194.134079] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6194.312234] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6204.387354] LustreError: 195179:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.409778] LustreError: 195179:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6209.516272] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6209.523830] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6209.529558] Lustre: Skipped 1 previous similar message [ 6214.635647] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6214.642179] Lustre: Skipped 5 previous similar messages [ 6215.047864] Lustre: server umount lustre-OST0001 complete [ 6219.744985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6219.771896] Lustre: Skipped 2 previous similar messages [ 6233.055410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6233.225156] Lustre: server umount lustre-MDT0000 complete [ 6236.693658] LustreError: 193337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781493 with bad export cookie 18153413276178859801 [ 6236.707347] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6236.715569] LustreError: 193337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6237.063976] Lustre: server umount lustre-MDT0001 complete [ 6264.033924] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 07:45:19 (1788781519) [ 6272.164383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6272.632700] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6272.712282] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6273.017317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6279.096923] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6287.975580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6288.112755] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6288.172918] Lustre: 199642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6292.532692] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6300.133882] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6308.592663] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6317.076671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6317.349463] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6323.391597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6324.744636] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6333.977468] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6341.331385] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6341.526888] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6341.775587] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6341.780177] Lustre: Skipped 2 previous similar messages [ 6348.973203] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6350.927872] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6356.972975] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6361.061386] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.064826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6361.067527] Lustre: Skipped 1 previous similar message [ 6361.070933] Lustre: Skipped 6 previous similar messages [ 6366.028873] Lustre: server umount lustre-OST0000 complete [ 6366.196343] LustreError: 203002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6366.202642] LustreError: 203002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 6386.660878] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6386.677367] LustreError: Skipped 2 previous similar messages [ 6391.816386] Lustre: server umount lustre-OST0001 complete [ 6396.192538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6396.201680] Lustre: Skipped 3 previous similar messages [ 6402.667929] Lustre: server umount lustre-MDT0000 complete [ 6406.359777] LustreError: 200387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781663 with bad export cookie 18153413276178861201 [ 6406.374072] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6406.376824] LustreError: 200387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6406.810162] Lustre: server umount lustre-MDT0001 complete [ 6436.514422] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 07:48:11 (1788781691) [ 6454.744591] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 6468.350698] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 6507.835200] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6508.037044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6508.059038] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6508.126365] Lustre: lustre-MDT0000: new disk, initializing [ 6508.203305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6508.229837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6513.432436] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6525.381912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6525.710282] Lustre: 207745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6525.810836] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6525.816992] Lustre: Skipped 1 previous similar message [ 6525.895384] Lustre: lustre-MDT0001: new disk, initializing [ 6525.985667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6525.995353] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6531.549312] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6543.118261] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6550.304758] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6557.977773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6558.179235] Lustre: lustre-OST0000: new disk, initializing [ 6558.182720] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6558.187797] Lustre: 209714:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6565.447398] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6566.458121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6566.490602] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6566.591192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6575.304328] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6581.584987] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6581.870174] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6586.676358] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6586.929782] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6592.485860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.489182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6592.509567] Lustre: Skipped 1 previous similar message [ 6592.540667] Lustre: Skipped 8 previous similar messages [ 6603.231273] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6603.461642] Lustre: server umount lustre-OST0000 complete [ 6622.175131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6622.453445] Lustre: server umount lustre-MDT0000 complete [ 6623.201740] LustreError: 207750:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.225685] LustreError: 207750:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6627.515948] LustreError: 207737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788781884 with bad export cookie 18153413276178862986 [ 6627.532461] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6627.536456] LustreError: 207737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6627.986741] Lustre: server umount lustre-MDT0001 complete [ 6637.856215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6643.706231] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6656.343926] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6662.154937] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6671.534570] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6677.278659] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6683.791722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6690.852976] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6700.138534] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6705.150574] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6709.906038] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6715.125130] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6715.416406] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6718.955897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6723.713469] Lustre: server umount lustre-OST0000 complete [ 6730.361742] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6735.534377] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6764.595286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6765.014246] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.021323] Lustre: Skipped 5 previous similar messages [ 6771.056156] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6778.745650] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6784.128093] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 6784.500328] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6784.510777] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6784.511877] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6784.534074] Lustre: Skipped 1 previous similar message [ 6789.603447] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6789.613951] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6789.638366] Lustre: Skipped 5 previous similar messages [ 6794.577346] Lustre: server umount lustre-OST0000 complete [ 6805.810524] Lustre: server umount lustre-MDT0000 complete [ 6809.478472] LustreError: 211917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782066 with bad export cookie 18153413276178863875 [ 6809.486536] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6809.492104] LustreError: 211917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6809.786943] Lustre: server umount lustre-MDT0001 complete [ 6818.927688] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 07:54:34 (1788782074) [ 6826.289176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6826.848245] LustreError: 217933:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6826.867133] LustreError: 217933:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6831.981829] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6841.030886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6846.047798] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6855.191581] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6861.351851] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6869.251510] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6876.376511] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6884.999472] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6890.107879] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6890.284036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6894.440821] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6894.641832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6902.965514] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6903.271618] Lustre: lustre-OST0001: new disk, initializing [ 6903.285919] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6903.296595] Lustre: 221303:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6910.097804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6910.140527] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6910.509850] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6913.561712] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 6924.682935] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6930.470972] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6930.822539] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6935.223834] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6935.470774] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6941.164240] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6941.175491] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6941.178331] LustreError: Skipped 1 previous similar message [ 6941.194393] Lustre: Skipped 17 previous similar messages [ 6943.977987] Lustre: server umount lustre-OST0001 complete [ 6951.673483] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6951.976658] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6957.528938] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6957.770352] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6989.861743] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6995.440802] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6995.452291] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6995.452625] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6995.468974] Lustre: Skipped 1 previous similar message [ 6996.901069] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7005.757350] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7007.797538] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7010.784158] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7010.792605] LustreError: Skipped 1 previous similar message [ 7016.439110] Lustre: server umount lustre-OST0001 complete [ 7040.455642] Lustre: server umount lustre-OST0000 complete [ 7046.119214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7046.151874] Lustre: Skipped 8 previous similar messages [ 7050.901289] Lustre: server umount lustre-MDT0000 complete [ 7055.053504] LustreError: 217915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782312 with bad export cookie 18153413276178864841 [ 7055.054866] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7055.078715] LustreError: 217915:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7061.548237] Lustre: server umount lustre-MDT0001 complete [ 7089.199352] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 07:59:04 (1788782344) [ 7096.757477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7097.200396] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7097.290831] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7103.056392] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7112.852798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7112.946619] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7112.989988] Lustre: 226552:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7118.567287] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7128.010938] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7134.930951] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7143.463503] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7143.765076] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7152.378132] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7161.816614] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7172.339823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7172.425711] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7181.250660] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7191.142529] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7193.782320] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7200.732634] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7201.029939] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7207.503052] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7207.763964] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7215.555441] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9b5504258800.ost_server_uuid 50 [ 7217.175776] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9b5504258800.ost_server_uuid in IDLE state after 0 sec [ 7218.433309] Lustre: Permanently deactivating lustre-OST0001 [ 7218.454731] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7223.068738] Lustre: Permanently reactivating lustre-OST0001 [ 7223.090871] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7223.104326] Lustre: Skipped 2 previous similar messages [ 7232.483627] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7232.487732] LustreError: Skipped 3 previous similar messages [ 7234.946750] Lustre: server umount lustre-OST0001 complete [ 7237.608632] LustreError: 228718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7237.635182] LustreError: 228718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7245.991386] Lustre: server umount lustre-OST0000 complete [ 7265.247281] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7265.480956] Lustre: server umount lustre-MDT0000 complete [ 7270.371822] LustreError: 226545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782527 with bad export cookie 18153413276178866129 [ 7270.378255] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7270.387497] LustreError: 226545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7271.133715] Lustre: server umount lustre-MDT0001 complete [ 7300.814554] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 08:02:36 (1788782556) [ 7309.544145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7309.891530] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7309.920534] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7309.927347] Lustre: Skipped 2 previous similar messages [ 7310.154657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7310.162714] Lustre: Skipped 9 previous similar messages [ 7315.910829] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7326.928282] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7327.058336] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7327.110676] Lustre: 233267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7333.349487] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7342.156737] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7350.745836] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7359.257345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7359.502078] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7359.540968] Lustre: Permanently deactivating lustre-OST0000 [ 7359.552280] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7365.171032] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7367.461927] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7377.142285] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7385.720745] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7385.820112] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7394.036297] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7394.300551] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7403.739462] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7406.064945] Lustre: Permanently reactivating lustre-OST0000 [ 7406.081553] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7406.089935] Lustre: Skipped 2 previous similar messages [ 7414.370812] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7414.374279] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 49 seconds. I think it's dead, and I am evicting it. exp 00000000cea8b449, cur 1788782671 deadline 1788782652 last 1788782622 [ 7414.378661] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7414.396220] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7414.472433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7425.023431] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7425.046100] LustreError: Skipped 2 previous similar messages [ 7426.721367] Lustre: server umount lustre-OST0001 complete [ 7437.919867] Lustre: server umount lustre-OST0000 complete [ 7448.724531] Lustre: server umount lustre-MDT0000 complete [ 7452.934735] LustreError: 236152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782709 with bad export cookie 18153413276178867802 [ 7452.937270] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7452.947809] LustreError: 236152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7453.474345] Lustre: server umount lustre-MDT0001 complete [ 7465.010471] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 7468.083507] Key type lgssc unregistered [ 7468.374242] LNet: 238151:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7468.401245] LNetError: 238151:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7469.480300] LNet: Removed LNI 192.168.204.125@tcp [ 7470.582267] Key type .llcrypt unregistered [ 7470.584738] Key type ._llcrypt unregistered [ 7497.991242] Key type ._llcrypt registered [ 7497.993560] Key type .llcrypt registered [ 7498.186754] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7499.617673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7499.647027] alg: No test for adler32 (adler32-zlib) [ 7500.727699] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 7500.974450] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 7502.615236] Key type lgssc registered [ 7504.097036] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7515.158928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7515.194752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7516.580224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7520.976072] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7526.881787] LustreError: 239616:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7526.901081] LustreError: 239616:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7530.584664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7530.710643] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7530.744977] Lustre: 239604:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7530.756339] Lustre: Permanently deactivating lustre-MDT0001 [ 7530.800929] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7531.358411] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7535.331829] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7535.345869] LustreError: 239647:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7535.373609] LustreError: 239647:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7535.382900] LustreError: 239647:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 1, failed: rc = -108 [ 7536.329943] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7544.134504] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7550.240285] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7557.459506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7557.844874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.970892] LustreError: 241458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7561.990945] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7564.067671] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7567.331643] LustreError: 241458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7572.453468] LustreError: 241673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.373846] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7576.552801] LustreError: 241673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7576.562838] LustreError: 241673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7580.893716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7581.083088] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7586.310517] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7586.319765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7588.907742] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7597.595582] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7601.589322] Lustre: Permanently reactivating lustre-MDT0001 [ 7601.624676] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7601.631564] Lustre: Skipped 2 previous similar messages [ 7605.884477] Lustre: lustre-MDT0001: Client c097a607-923a-42c9-b54c-d7e14c745360 (at 192.168.204.25@tcp) reconnecting [ 7613.236271] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7615.144691] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7621.195402] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7621.515531] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7626.400303] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7626.704221] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7629.768087] Lustre: Permanently deactivating lustre-MDT0001 [ 7629.792476] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7629.797345] Lustre: Skipped 2 previous similar messages [ 7637.613260] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7647.714513] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7647.720098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7647.723379] Lustre: Skipped 3 previous similar messages [ 7647.729651] Lustre: Skipped 3 previous similar messages [ 7651.840489] Lustre: server umount lustre-MDT0000 complete [ 7652.836476] LustreError: 243178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7652.858051] LustreError: 243178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7656.351061] LustreError: 239596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788782913 with bad export cookie 13457160959857107578 [ 7656.365991] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7656.680921] Lustre: server umount lustre-MDT0001 complete [ 7663.161233] Lustre: server umount lustre-OST0000 complete [ 7676.930431] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 08:08:52 (1788782932) [ 7688.488628] Lustre: server umount lustre-OST0001 complete [ 7696.031409] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 7707.310379] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7748.568594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7748.877056] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7748.890120] Lustre: Skipped 2 previous similar messages [ 7748.927701] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7749.088045] Lustre: lustre-MDT0000: new disk, initializing [ 7749.210884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7749.232269] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7754.558549] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7765.854205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7765.932815] Lustre: 247875:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7765.978454] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7765.983593] Lustre: Skipped 1 previous similar message [ 7766.086270] Lustre: lustre-MDT0001: new disk, initializing [ 7766.168492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7766.213155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7766.223761] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7771.238146] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7782.152995] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7789.024425] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7796.327181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7796.664802] Lustre: lustre-OST0000: new disk, initializing [ 7796.672538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7796.682829] Lustre: 249848:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7796.758450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7802.472930] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7802.497640] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7802.667904] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7803.996933] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7816.946591] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7820.304298] LustreError: 247883:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7830.351148] LustreError: 247883:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7830.389321] LustreError: 250625:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7831.118398] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7831.393707] Lustre: lustre-OST0001: new disk, initializing [ 7831.396878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7831.401721] Lustre: 250966:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7831.550873] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7839.241460] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 7840.403148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7840.415170] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7840.448804] LustreError: 250625:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7840.591446] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7850.442902] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7852.284113] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7856.104050] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7856.130429] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.150279] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7856.158825] Lustre: Skipped 1 previous similar message [ 7860.843684] Lustre: server umount lustre-OST0001 complete [ 7861.230060] LustreError: 251391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7861.258909] LustreError: 251391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7879.648224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7879.654982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7879.657451] LustreError: Skipped 2 previous similar messages [ 7879.658486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7879.672042] Lustre: Skipped 2 previous similar messages [ 7881.697372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7881.708558] Lustre: Skipped 2 previous similar messages [ 7885.331192] Lustre: server umount lustre-OST0000 complete [ 7889.379313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7889.392874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7889.414665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7902.179737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7902.188947] Lustre: Skipped 5 previous similar messages [ 7903.200033] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7903.414718] Lustre: server umount lustre-MDT0000 complete [ 7906.279438] LustreError: 249021:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7906.293966] LustreError: 249021:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7907.387745] LustreError: 247868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783164 with bad export cookie 13457160959857110175 [ 7907.397302] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7907.403853] LustreError: 247868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7913.911769] Lustre: server umount lustre-MDT0001 complete [ 7924.820720] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 7927.445780] Key type lgssc unregistered [ 7927.702704] LNet: 252999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7927.708035] LNetError: 252999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7927.720782] LNet: Removed LNI 192.168.204.125@tcp [ 7928.615356] Key type .llcrypt unregistered [ 7928.621226] Key type ._llcrypt unregistered [ 7970.197472] Key type ._llcrypt registered [ 7970.199966] Key type .llcrypt registered [ 7970.357961] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 7971.900320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7971.941592] alg: No test for adler32 (adler32-zlib) [ 7973.079851] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 7973.497909] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 7975.264253] Key type lgssc registered [ 7976.655915] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7984.927516] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7984.950640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7986.262856] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7986.284240] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7986.466812] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7991.908758] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8001.507588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8001.567658] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8001.603395] Lustre: 255219:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8001.899766] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8006.330612] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8014.571042] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8021.350669] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8028.848585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8029.034843] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8029.265404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8036.129606] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8038.056037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8045.222183] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8099.809121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8099.826370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.839947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8104.420556] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8104.433073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.461144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8105.637701] Lustre: server umount lustre-OST0000 complete [ 8110.753727] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8124.239903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8124.709240] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8125.740900] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8125.766270] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8130.970893] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8130.989094] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8131.825309] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8139.890065] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8182.240188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8182.247939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8182.251444] LustreError: Skipped 1 previous similar message [ 8182.253885] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8182.267581] Lustre: Skipped 1 previous similar message [ 8182.271082] Lustre: Skipped 1 previous similar message [ 8185.924430] Lustre: server umount lustre-OST0000 complete [ 8191.461973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8191.469431] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.493062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8191.498122] Lustre: Skipped 1 previous similar message [ 8197.602967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8197.607880] Lustre: Skipped 3 previous similar messages [ 8204.256250] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8204.508113] Lustre: server umount lustre-MDT0000 complete [ 8208.336312] LustreError: 255210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783465 with bad export cookie 4044932749052212768 [ 8208.343586] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8208.350921] LustreError: 255210:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8208.702819] Lustre: server umount lustre-MDT0001 complete [ 8220.250855] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8223.270395] Key type lgssc unregistered [ 8223.675725] LNet: 260643:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8223.687317] LNetError: 260643:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8223.706979] LNet: Removed LNI 192.168.204.125@tcp [ 8224.550379] Key type .llcrypt unregistered [ 8224.555737] Key type ._llcrypt unregistered [ 8248.253363] Key type ._llcrypt registered [ 8248.258280] Key type .llcrypt registered [ 8248.385475] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8249.301117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8249.361760] alg: No test for adler32 (adler32-zlib) [ 8250.514293] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 8250.837321] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8252.543183] Key type lgssc registered [ 8253.704056] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8262.443953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8262.456814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8263.825140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8268.618600] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8273.380374] LustreError: 262058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8273.399556] LustreError: 262058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8278.258349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8278.506241] LustreError: 262059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8278.724449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8283.761893] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8293.114539] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8300.184922] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8309.855376] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8310.302893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8318.300461] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8326.412269] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8333.869984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8364.517029] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8364.519847] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8364.524367] LustreError: Skipped 1 previous similar message [ 8364.530653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8364.539539] Lustre: Skipped 1 previous similar message [ 8364.557197] Lustre: Skipped 1 previous similar message [ 8366.556222] Lustre: server umount lustre-OST0000 complete [ 8371.168780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8371.183094] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8371.194159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8374.757598] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8374.759092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8374.783524] Lustre: Skipped 1 previous similar message [ 8377.279623] Lustre: server umount lustre-MDT0000 complete [ 8379.879891] LustreError: 262059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8379.898287] LustreError: 262059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8381.676410] LustreError: 262040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783638 with bad export cookie 15556983104791738073 [ 8381.680114] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8381.695684] LustreError: 262040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8382.254567] Lustre: server umount lustre-MDT0001 complete [ 8393.703957] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8397.158866] Key type lgssc unregistered [ 8397.558795] LNet: 266388:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8397.572028] LNetError: 266388:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8397.592271] LNet: Removed LNI 192.168.204.125@tcp [ 8398.488872] Key type .llcrypt unregistered [ 8398.491162] Key type ._llcrypt unregistered [ 8416.289393] Key type ._llcrypt registered [ 8416.291518] Key type .llcrypt registered [ 8416.439463] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8417.463811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8417.504640] alg: No test for adler32 (adler32-zlib) [ 8418.690548] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 8419.009872] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8420.703198] Key type lgssc registered [ 8422.143898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8431.021196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8431.052024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8432.809074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8438.152056] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8442.338956] LustreError: 267606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8442.365828] LustreError: 267606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8447.284977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8447.460229] LustreError: 267607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8447.735188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8453.035552] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8461.853446] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8469.181635] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8476.413711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8476.908420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8477.971552] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8484.009296] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8491.596731] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8507.873696] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8507.876634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8507.881709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8507.899306] Lustre: Skipped 1 previous similar message [ 8511.128867] Lustre: server umount lustre-OST0000 complete [ 8518.115380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8518.127708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8518.152850] Lustre: Skipped 1 previous similar message [ 8518.155307] Lustre: Skipped 1 previous similar message [ 8522.043661] Lustre: server umount lustre-MDT0000 complete [ 8523.238570] LustreError: 267607:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8523.260312] LustreError: 267607:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8526.369360] LustreError: 267586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783783 with bad export cookie 1735987768919265238 [ 8526.378458] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8526.389085] LustreError: 267586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8526.994811] Lustre: server umount lustre-MDT0001 complete [ 8539.531859] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8542.935269] Key type lgssc unregistered [ 8543.298830] LNet: 271284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8543.319976] LNetError: 271284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8543.338534] LNet: Removed LNI 192.168.204.125@tcp [ 8544.257165] Key type .llcrypt unregistered [ 8544.260062] Key type ._llcrypt unregistered [ 8569.639305] Key type ._llcrypt registered [ 8569.641710] Key type .llcrypt registered [ 8569.743933] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8570.891033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8570.901483] alg: No test for adler32 (adler32-zlib) [ 8571.948807] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 8572.193758] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8573.943629] Key type lgssc registered [ 8575.356945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8584.553987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8584.564946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8586.059555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.332919] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8596.451986] LustreError: 272715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8596.509795] LustreError: 272715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8601.582703] LustreError: 272716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8606.698861] LustreError: 272715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8607.787551] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8608.536208] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8614.549735] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8623.334238] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8630.511656] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8638.494992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8638.988243] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8645.823319] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8653.787732] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8656.457022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8684.002472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8684.014362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.041268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8687.077569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8687.079969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8687.090165] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8687.122984] Lustre: Skipped 1 previous similar message [ 8689.430485] Lustre: server umount lustre-OST0000 complete [ 8695.267755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8695.277455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8695.300968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8697.317252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8697.318462] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8700.426134] Lustre: server umount lustre-MDT0000 complete [ 8702.449204] LustreError: 272715:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8702.484271] LustreError: 272715:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8704.584897] LustreError: 272697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788783961 with bad export cookie 10444158720608147880 [ 8704.601222] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8704.613280] LustreError: 272697:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8704.941884] Lustre: server umount lustre-MDT0001 complete [ 8718.418882] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8722.320205] Key type lgssc unregistered [ 8722.655583] LNet: 277023:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8722.673161] LNetError: 277023:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8722.695721] LNet: Removed LNI 192.168.204.125@tcp [ 8723.965809] Key type .llcrypt unregistered [ 8723.971399] Key type ._llcrypt unregistered [ 8744.404520] Key type ._llcrypt registered [ 8744.408464] Key type .llcrypt registered [ 8744.543251] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8745.678461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8745.700481] alg: No test for adler32 (adler32-zlib) [ 8746.763990] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 8746.992247] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8748.679502] Key type lgssc registered [ 8750.029114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8758.728481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8758.753132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8760.360216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8765.273964] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8770.536159] LustreError: 278258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8770.573654] LustreError: 278258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8774.237126] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8774.669436] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8779.857560] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8788.019775] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8793.947381] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8802.110208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8802.530756] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.612461] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8809.726073] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8817.646548] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8835.564330] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8835.584459] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.595102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8835.625494] Lustre: Skipped 1 previous similar message [ 8835.662961] Lustre: Skipped 1 previous similar message [ 8836.871898] Lustre: server umount lustre-OST0000 complete [ 8845.793083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8845.797464] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8845.808364] Lustre: Skipped 1 previous similar message [ 8845.815720] Lustre: Skipped 1 previous similar message [ 8849.164780] Lustre: server umount lustre-MDT0000 complete [ 8850.922628] LustreError: 278263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8850.940971] LustreError: 278263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8852.920234] LustreError: 278223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784109 with bad export cookie 495513715294663022 [ 8852.929031] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8852.942576] LustreError: 278223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8853.585198] Lustre: server umount lustre-MDT0001 complete [ 8866.444905] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 8869.904938] Key type lgssc unregistered [ 8870.276051] LNet: 281958:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8870.288597] LNetError: 281958:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8870.310426] LNet: Removed LNI 192.168.204.125@tcp [ 8871.201943] Key type .llcrypt unregistered [ 8871.206953] Key type ._llcrypt unregistered [ 8885.307329] Key type ._llcrypt registered [ 8885.310897] Key type .llcrypt registered [ 8885.476595] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8903.921241] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 8919.623472] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 8920.611349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8920.830245] alg: No test for adler32 (adler32-zlib) [ 8922.017350] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 8922.363698] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 8924.023188] Key type lgssc registered [ 8925.184783] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8966.351959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8966.388205] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8967.726548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8967.771598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8967.860459] Lustre: lustre-MDT0000: new disk, initializing [ 8967.974561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8967.994460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8973.487291] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 8985.387438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8985.513656] Lustre: 285508:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8985.570110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8985.577539] Lustre: Skipped 1 previous similar message [ 8985.695705] Lustre: lustre-MDT0001: new disk, initializing [ 8985.808987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8985.875879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8985.886725] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8991.032987] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9002.486813] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9009.092288] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9017.944989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9018.191957] Lustre: lustre-OST0000: new disk, initializing [ 9018.199978] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9018.228476] Lustre: 287477:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9018.355207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9026.237276] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9028.196299] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9028.217542] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9028.377315] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9040.024614] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9046.558856] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9046.972245] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9053.360672] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9053.619834] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9059.297143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9059.303635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9059.312376] Lustre: Skipped 1 previous similar message [ 9059.331350] Lustre: Skipped 1 previous similar message [ 9062.413656] Lustre: server umount lustre-OST0000 complete [ 9068.001974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9068.009536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9068.036630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9069.538925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9069.543784] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9073.754742] Lustre: server umount lustre-MDT0000 complete [ 9078.136331] LustreError: 286313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784335 with bad export cookie 16670604531108387612 [ 9078.138349] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9078.151789] LustreError: 286313:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9078.572392] Lustre: server umount lustre-MDT0001 complete [ 9087.794636] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 08:32:23 (1788784343) [ 9094.979804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.562642] LustreError: 289891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9095.587381] LustreError: 289891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9095.648993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9099.750918] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9100.768795] LustreError: 289892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9105.896261] LustreError: 289891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9109.965293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9110.388830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9114.350087] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9122.019860] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9128.929643] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9135.631429] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9136.052973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9142.213927] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9149.415224] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9165.792786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9165.794462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9165.807525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9165.807533] Lustre: Skipped 1 previous similar message [ 9165.810986] LustreError: Skipped 1 previous similar message [ 9165.820659] Lustre: Skipped 1 previous similar message [ 9168.105697] Lustre: server umount lustre-OST0000 complete [ 9176.039241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9176.055177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9176.057745] Lustre: Skipped 1 previous similar message [ 9178.628843] Lustre: server umount lustre-MDT0000 complete [ 9181.152835] LustreError: 289892:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9181.171871] LustreError: 289892:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9182.292753] LustreError: 289872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784439 with bad export cookie 16670604531108388487 [ 9182.312428] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9182.783057] Lustre: server umount lustre-MDT0001 complete [ 9195.311548] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 9198.213415] Key type lgssc unregistered [ 9198.535217] LNet: 293445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9198.540393] LNetError: 293445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9198.560116] LNet: Removed LNI 192.168.204.125@tcp [ 9199.468796] Key type .llcrypt unregistered [ 9199.470395] Key type ._llcrypt unregistered [ 9227.860983] Key type ._llcrypt registered [ 9227.862972] Key type .llcrypt registered [ 9227.951397] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 9242.796252] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9243.799359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9243.963894] alg: No test for adler32 (adler32-zlib) [ 9245.046634] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 9245.314537] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 9247.031221] Key type lgssc registered [ 9248.190145] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9294.839602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9294.858679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9296.191284] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9296.239804] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9296.317714] Lustre: lustre-MDT0000: new disk, initializing [ 9296.391149] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9296.405987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9301.109626] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9311.353677] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9311.491789] Lustre: 297271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9311.559512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9311.563741] Lustre: Skipped 1 previous similar message [ 9311.628157] Lustre: lustre-MDT0001: new disk, initializing [ 9311.705748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9311.743108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9311.757500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9316.277353] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9325.087752] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9331.146870] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9338.859786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9339.245524] Lustre: lustre-OST2710: new disk, initializing [ 9339.249809] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9339.261160] Lustre: 299240:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9339.364281] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9346.456837] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9348.654844] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9348.668782] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9348.783908] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9356.465938] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9364.015131] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9364.152165] Lustre: lustre-OST03e8: new disk, initializing [ 9364.155746] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9364.162705] Lustre: 300305:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9364.251395] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9370.606600] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9371.212730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9371.227676] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9371.320510] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9380.534631] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9389.712853] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9390.054661] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9394.836673] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9395.010680] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9399.086592] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9399.300728] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9403.604582] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9403.826356] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9409.011748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9409.020560] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9409.048937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9411.358077] Lustre: server umount lustre-MDT0000 complete [ 9412.590201] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9412.614062] Lustre: Skipped 1 previous similar message [ 9415.554971] LustreError: 299238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784672 with bad export cookie 16836883046522948056 [ 9415.556566] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9415.575249] LustreError: 299238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9416.455432] Lustre: server umount lustre-MDT0001 complete [ 9430.777651] Lustre: server umount lustre-OST2710 complete [ 9433.056432] Lustre: 295068:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788784674/real 1788784674] req@0000000090ad6b76 x1875676498039296/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788784690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9433.080572] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9433.095930] Lustre: Skipped 1 previous similar message [ 9434.058070] Lustre: server umount lustre-OST03e8 complete [ 9452.937982] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [ 9468.625328] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9508.549947] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9508.800834] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9508.833695] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9508.913358] Lustre: lustre-MDT0000: new disk, initializing [ 9508.981292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9509.008377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9513.871976] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9523.421618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9523.506232] Lustre: 305579:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9523.547629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9523.552684] Lustre: Skipped 1 previous similar message [ 9523.615619] Lustre: lustre-MDT0001: new disk, initializing [ 9523.698565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9523.737861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9523.752902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9527.848341] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9538.532366] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9546.065148] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9554.519994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9554.789711] Lustre: lustre-OST0000: new disk, initializing [ 9554.799102] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9554.806633] Lustre: 307546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9554.880982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9562.327806] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9564.219123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9564.236572] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9564.389213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9573.124071] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9578.564390] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9578.840862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9583.198647] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9583.351843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9590.252210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9590.258951] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9590.264308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9590.288989] Lustre: Skipped 1 previous similar message [ 9590.309582] Lustre: Skipped 1 previous similar message [ 9591.752266] Lustre: server umount lustre-OST0000 complete [ 9600.483672] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9600.490697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9600.496227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9600.512919] Lustre: Skipped 1 previous similar message [ 9600.548288] LustreError: Skipped 1 previous similar message [ 9602.426142] Lustre: server umount lustre-MDT0000 complete [ 9605.608121] LustreError: 306741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9605.635489] LustreError: 306741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9606.745816] LustreError: 305571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784863 with bad export cookie 16836883046522949729 [ 9606.749040] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9606.755945] LustreError: 305571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9607.159587] Lustre: server umount lustre-MDT0001 complete [ 9618.413624] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 08:41:13 (1788784873) [ 9620.170093] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9622.108697] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 08:41:17 (1788784877) [ 9625.997461] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9659.274293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9659.480224] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9659.500950] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9659.781556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9665.102250] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9674.937221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9675.105733] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9675.163891] Lustre: 311408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9680.895850] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9703.607055] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9703.881799] Lustre: Denying initial registration attempt from nid 192.168.204.125@tcp, specified as failover [ 9703.895594] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9703.911049] LustreError: 312862:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9703.926523] LustreError: 312862:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9703.940770] LustreError: 312862:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9704.005112] Lustre: server umount lustre-OST0000 complete [ 9704.008505] LustreError: 312862:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9711.585032] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9711.592567] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9711.607878] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9711.614293] Lustre: Skipped 1 previous similar message [ 9717.276436] Lustre: server umount lustre-MDT0000 complete [ 9718.753093] LustreError: 311413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9718.781817] LustreError: 311413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9721.108687] LustreError: 311398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788784978 with bad export cookie 16836883046522950611 [ 9721.114125] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9721.121214] LustreError: 311398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9721.558833] Lustre: server umount lustre-MDT0001 complete [ 9733.267612] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 9736.346651] Key type lgssc unregistered [ 9736.674362] LNet: 313980:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9736.683263] LNetError: 313980:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9736.696981] LNet: Removed LNI 192.168.204.125@tcp [ 9737.524770] Key type .llcrypt unregistered [ 9737.528500] Key type ._llcrypt unregistered [ 9749.833303] Key type ._llcrypt registered [ 9749.835623] Key type .llcrypt registered [ 9749.937588] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9751.141928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9751.204259] alg: No test for adler32 (adler32-zlib) [ 9752.364245] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 9752.728130] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 9754.399204] Key type lgssc registered [ 9755.541655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9803.283564] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9811.093145] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9811.136402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9812.385415] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9812.407778] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9812.569573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9816.764374] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9825.553189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9825.657664] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9825.701593] Lustre: 316746:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9825.957249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9829.943435] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9837.680374] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9844.608511] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9866.442374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9866.634473] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9866.831093] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9872.106744] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9880.557645] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9886.693753] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9886.715585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9886.731025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9887.206619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9887.228205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9887.244554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9887.251465] Lustre: Skipped 1 previous similar message [ 9889.938798] Lustre: server umount lustre-OST0000 complete [ 9897.441063] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9897.448557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9897.466965] Lustre: Skipped 1 previous similar message [ 9897.483757] Lustre: Skipped 1 previous similar message [ 9900.796675] Lustre: server umount lustre-MDT0000 complete [ 9905.014324] LustreError: 316738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785162 with bad export cookie 1719209987745513163 [ 9905.025360] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9905.027971] LustreError: 316738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9905.498487] Lustre: server umount lustre-MDT0001 complete [ 9916.114911] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [ 9918.609833] Key type lgssc unregistered [ 9918.905806] LNet: 320318:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9918.910351] LNetError: 320318:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9918.935274] LNet: Removed LNI 192.168.204.125@tcp [ 9919.796219] Key type .llcrypt unregistered [ 9919.797938] Key type ._llcrypt unregistered [ 9945.193471] Key type ._llcrypt registered [ 9945.195459] Key type .llcrypt registered [ 9945.324189] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [ 9946.750441] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9946.797862] alg: No test for adler32 (adler32-zlib) [ 9947.834488] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [ 9948.045839] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [ 9949.719176] Key type lgssc registered [ 9950.971671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9959.300319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9959.319757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9960.932988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9966.014952] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9970.155638] LustreError: 321733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9970.170830] LustreError: 321733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9975.275534] LustreError: 321732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9975.335826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9975.649352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9980.469227] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [ 9987.993317] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9994.255813] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10003.373488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10003.645483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10009.907235] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10018.857890] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10030.576075] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10030.594511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10030.598853] Lustre: Skipped 2 previous similar messages [10030.625135] Lustre: Skipped 2 previous similar messages [10031.184725] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [10034.053630] Lustre: server umount lustre-MDT0000 complete [10035.684770] LustreError: 321729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10035.716682] LustreError: 321729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10039.290699] LustreError: 321714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785296 with bad export cookie 9160170463962363791 [10039.308836] LustreError: 321714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10039.310604] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10040.802278] LustreError: 321729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10040.808924] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10040.835867] LustreError: 321729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10040.881918] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10044.839710] LustreError: 321728:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10044.841567] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10044.857626] LustreError: 321728:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10044.891426] Lustre: Skipped 1 previous similar message [10045.873064] Lustre: server umount lustre-MDT0001 complete [10052.887131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10061.789768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10064.837926] LustreError: 323564:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10064.968472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10065.045960] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10070.246519] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10075.118096] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10081.341204] LustreError: 325324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10081.347459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.356573] LustreError: 325324:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10081.910459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10086.908537] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10087.117439] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10096.285871] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10098.182586] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10106.310363] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10108.129642] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10116.065142] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10116.079486] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10116.102522] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10119.927499] Lustre: server umount lustre-OST0000 complete [10127.841474] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10127.843183] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10127.850406] Lustre: Skipped 1 previous similar message [10127.858241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10127.861307] Lustre: Skipped 3 previous similar messages [10130.395259] Lustre: server umount lustre-MDT0000 complete [10132.961737] LustreError: 325327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10132.973752] LustreError: 325327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10134.621727] LustreError: 324515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785391 with bad export cookie 9160170463962370945 [10134.634261] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10134.637789] LustreError: 324515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10135.047559] Lustre: server umount lustre-MDT0001 complete [10150.853217] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10154.081314] Key type lgssc unregistered [10154.429654] LNet: 327998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10154.448307] LNetError: 327998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10154.473766] LNet: Removed LNI 192.168.204.125@tcp [10155.961429] Key type .llcrypt unregistered [10155.967795] Key type ._llcrypt unregistered [10184.280735] Key type ._llcrypt registered [10184.290347] Key type .llcrypt registered [10184.401316] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [10185.091350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10185.111127] alg: No test for adler32 (adler32-zlib) [10186.103714] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [10186.277180] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [10187.967248] Key type lgssc registered [10189.096561] Lustre: Echo OBD driver; http://www.lustre.org/ [10196.600237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10196.616808] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10197.992315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10202.328922] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10207.207485] LustreError: 329411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10207.246544] LustreError: 329411:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10211.211850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10211.658079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10215.993942] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10223.598588] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10229.018571] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10236.578128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10236.600121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10236.607307] Lustre: Skipped 1 previous similar message [10238.805304] Lustre: server umount lustre-MDT0000 complete [10242.018924] LustreError: 331011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10242.029794] LustreError: 331011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10242.544557] LustreError: 329393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785499 with bad export cookie 12237868838246624749 [10242.550781] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10242.556439] LustreError: 329393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10242.944994] Lustre: server umount lustre-MDT0001 complete [10257.194174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10257.481963] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10257.527991] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10257.745277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10262.109669] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10271.641408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10271.729231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10271.764305] Lustre: 331998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10272.066833] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10277.467376] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10286.693469] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10293.970672] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10305.652366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10305.844523] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10305.856263] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10305.864137] LustreError: 333883:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10305.869644] LustreError: 333883:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10305.875332] LustreError: 333883:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10305.957828] Lustre: server umount lustre-OST0000 complete [10305.961074] LustreError: 333883:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10317.570074] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10317.902976] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10318.105474] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10323.628793] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10325.305849] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10333.336259] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10341.982523] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10342.055234] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10342.151941] Lustre: lustre-OST0001: new disk, initializing [10342.158070] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10342.173974] Lustre: 335425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10342.278136] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10347.645592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10347.653586] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10347.757779] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10348.835250] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10357.618466] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10363.374817] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10363.377907] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10363.389168] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10363.408066] Lustre: Skipped 2 previous similar messages [10363.424961] Lustre: Skipped 1 previous similar message [10368.492210] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10368.502359] Lustre: Skipped 1 previous similar message [10373.607762] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10373.612202] Lustre: Skipped 1 previous similar message [10375.136969] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10375.261875] Lustre: server umount lustre-OST0001 complete [10378.731590] LustreError: 334351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10378.760601] LustreError: 334351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10379.756044] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10379.767815] LustreError: Skipped 1 previous similar message [10379.776470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10379.797106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10383.854510] LustreError: 334350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10383.877125] LustreError: 334350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10385.555738] Lustre: server umount lustre-OST0000 complete [10389.984467] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10389.988110] LustreError: Skipped 1 previous similar message [10389.995918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10390.007154] Lustre: Skipped 1 previous similar message [10390.012142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10390.014959] Lustre: Skipped 3 previous similar messages [10394.087145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10395.681653] Lustre: server umount lustre-MDT0000 complete [10399.201330] LustreError: 333189:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10399.212206] LustreError: 333189:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10399.488867] LustreError: 331989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785656 with bad export cookie 12237868838246625302 [10399.492664] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10399.495406] LustreError: 331989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10399.885414] Lustre: server umount lustre-MDT0001 complete [10411.504654] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10414.587425] Key type lgssc unregistered [10414.899601] LNet: 337384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10414.917968] LNetError: 337384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10415.974729] LNet: Removed LNI 192.168.204.125@tcp [10416.690971] Key type .llcrypt unregistered [10416.694399] Key type ._llcrypt unregistered [10496.806464] Key type ._llcrypt registered [10496.815610] Key type .llcrypt registered [10496.954112] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [10512.788493] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [10514.264280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10514.296277] alg: No test for adler32 (adler32-zlib) [10515.444134] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [10515.678180] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [10517.407243] Key type lgssc registered [10518.614688] Lustre: Echo OBD driver; http://www.lustre.org/ [10552.173255] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10552.200485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.384182] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10553.423458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10553.487466] Lustre: lustre-MDT0000: new disk, initializing [10553.553482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10553.564302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10557.354081] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10567.052871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10567.172839] Lustre: 342317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10567.241639] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10567.247751] Lustre: Skipped 1 previous similar message [10567.310579] Lustre: lustre-MDT0001: new disk, initializing [10567.382152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10567.415217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10567.425991] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10571.272771] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10579.254078] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10584.896151] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10591.211622] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10591.516195] Lustre: lustre-OST0000: new disk, initializing [10591.519779] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10591.526099] Lustre: 344287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10591.654486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10597.439855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10597.454691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10597.607765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10597.808451] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10607.411553] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10612.746397] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10613.051745] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10618.122145] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10618.388580] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10623.458970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10623.468120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10623.469818] LustreError: Skipped 1 previous similar message [10623.474934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10623.503215] Lustre: Skipped 1 previous similar message [10626.884925] Lustre: server umount lustre-OST0000 complete [10633.702166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10633.714336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10633.721405] Lustre: Skipped 2 previous similar messages [10637.550373] Lustre: server umount lustre-MDT0000 complete [10641.217051] LustreError: 342307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788785898 with bad export cookie 5159134513095178860 [10641.222554] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10641.229746] LustreError: 342307:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10641.639328] Lustre: server umount lustre-MDT0001 complete [10652.030742] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 08:58:27 (1788785907) [10661.252961] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 08:58:36 (1788785916) [10670.213204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10670.741825] LustreError: 347042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10670.760268] LustreError: 347042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10670.835593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10675.773804] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10676.195107] LustreError: 347043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.292808] LustreError: 347042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10685.421852] LustreError: 347043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10685.557730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10685.892329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10691.085887] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10699.128769] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10705.684413] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10712.751390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10713.111608] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10718.576596] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10726.048698] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10736.005875] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10737.629439] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10739.188948] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10740.949401] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10742.971553] Lustre: Failing over lustre-MDT0000 [10743.394702] Lustre: server umount lustre-MDT0000 complete [10745.832253] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10745.835916] LustreError: 349534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10745.852419] Lustre: Skipped 2 previous similar messages [10745.878058] LustreError: 349534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10756.065369] LustreError: 347042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10756.079366] LustreError: 347042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10761.188428] Lustre: 340542:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788786002/real 1788786002] req@0000000079d21b9a x1875677829730432/t0(0) o400->MGC192.168.204.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788786018 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10761.207535] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10762.217398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10771.425409] LustreError: 340538:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e45f7f62 x1875677829735040/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10771.720896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10771.765555] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10773.561952] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10776.293638] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10777.081102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10777.116218] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10777.177472] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10787.366546] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10789.140963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10793.432898] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10795.582698] Lustre: Failing over lustre-MDT0000 [10795.968542] Lustre: server umount lustre-MDT0000 complete [10797.540848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10797.544720] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10797.559564] LustreError: 347039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10797.559576] LustreError: 347039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [10797.563641] Lustre: Skipped 2 previous similar messages [10813.920657] Lustre: 340540:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788786054/real 1788786054] req@000000006364ffc0 x1875677829751040/t0(0) o400->MGC192.168.204.125@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788786070 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10813.944754] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10815.094785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10823.138867] LustreError: 340538:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003948d639 x1875677829756160/t0(0) o250->MGC192.168.204.125@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10823.576902] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10823.635584] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10823.772546] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10827.514077] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10828.801626] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10828.810951] Lustre: Skipped 2 previous similar messages [10828.841460] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10828.892882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10836.945877] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10838.665497] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10849.251994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.262306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10849.263443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10849.267679] Lustre: Skipped 3 previous similar messages [10849.279557] Lustre: Skipped 2 previous similar messages [10851.449166] Lustre: server umount lustre-MDT0000 complete [10854.372492] LustreError: 347043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10854.401984] LustreError: 347043:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 28 previous similar messages [10855.463546] LustreError: 347776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786112 with bad export cookie 5159134513095181576 [10855.467855] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10855.475678] LustreError: 347776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10855.747528] Lustre: server umount lustre-MDT0001 complete [10870.502211] Lustre: server umount lustre-OST0000 complete [10887.173560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.691835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10893.061798] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10902.572415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10907.990451] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10918.186856] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10924.769091] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10932.466937] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10932.647229] LustreError: 355217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10932.679069] LustreError: 355217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10937.853492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10939.646547] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [10947.734550] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10955.610827] Lustre: server umount lustre-OST0000 complete [10963.427130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10963.432758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10963.440593] Lustre: Skipped 1 previous similar message [10963.453588] Lustre: Skipped 1 previous similar message [10968.545213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10968.561417] Lustre: Skipped 1 previous similar message [10970.723105] Lustre: server umount lustre-MDT0000 complete [10974.692850] Lustre: 340542:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788786215/real 1788786215] req@00000000570b39a0 x1875677829801856/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788786231 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10975.594516] LustreError: 355238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786232 with bad export cookie 5159134513095182318 [10975.603778] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10975.618126] LustreError: 355238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10976.480751] Lustre: server umount lustre-MDT0001 complete [10988.022121] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [10990.924905] Key type lgssc unregistered [10991.275459] LNet: 356926:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10991.284854] LNetError: 356926:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10991.316120] LNet: Removed LNI 192.168.204.125@tcp [10992.264169] Key type .llcrypt unregistered [10992.274184] Key type ._llcrypt unregistered [11018.511541] Key type ._llcrypt registered [11018.515309] Key type .llcrypt registered [11018.608241] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [11019.498230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11019.554511] alg: No test for adler32 (adler32-zlib) [11020.604871] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [11020.798320] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [11022.471182] Key type lgssc registered [11023.562504] Lustre: Echo OBD driver; http://www.lustre.org/ [11031.090819] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11031.112383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11032.581395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11038.492075] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11042.785805] LustreError: 358440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.801490] LustreError: 358440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11047.722533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11047.907064] LustreError: 358441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11048.252792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11053.720990] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11061.269934] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11067.586386] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11074.647244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11075.092659] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11077.197422] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11081.640196] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11089.527674] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11104.225395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11104.228830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11104.240544] Lustre: Skipped 2 previous similar messages [11104.245639] Lustre: Skipped 1 previous similar message [11106.159943] Lustre: server umount lustre-MDT0000 complete [11109.356040] LustreError: 358441:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11109.376341] LustreError: 358441:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11109.776230] LustreError: 358421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786366 with bad export cookie 15845391760925547343 [11109.783695] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11109.800493] LustreError: 358421:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11110.409154] Lustre: server umount lustre-MDT0001 complete [11125.844492] Lustre: server umount lustre-OST0000 complete [11138.956289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11139.427468] LustreError: 362177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11139.523875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11144.391195] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11144.680197] LustreError: 362178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11153.077522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11153.389273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11158.078166] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11166.584681] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11172.375820] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11180.802310] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11181.263607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11184.304812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11187.936281] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11195.108542] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11209.189582] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11209.191034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11209.217965] Lustre: Skipped 1 previous similar message [11209.243305] Lustre: Skipped 2 previous similar messages [11213.472252] Lustre: server umount lustre-MDT0000 complete [11214.305950] LustreError: 362178:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11214.322255] LustreError: 362178:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11218.119982] LustreError: 362157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786475 with bad export cookie 15845391760925548456 [11218.132127] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11218.137887] LustreError: 362157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11218.651486] Lustre: server umount lustre-MDT0001 complete [11234.186451] Lustre: server umount lustre-OST0000 complete [11252.174273] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.765567] LustreError: 366088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.840508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11257.314880] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11265.640070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11270.222889] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11278.185901] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11284.902145] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11291.636040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11292.065053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11292.076543] Lustre: Skipped 1 previous similar message [11297.224307] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11299.666421] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11307.844912] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11315.685912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11315.691649] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11315.710363] Lustre: Skipped 2 previous similar messages [11315.725510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11319.062764] Lustre: server umount lustre-OST0000 complete [11325.921402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11325.926330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11325.937488] Lustre: Skipped 1 previous similar message [11325.962090] Lustre: Skipped 2 previous similar messages [11329.059195] Lustre: server umount lustre-MDT0000 complete [11331.045866] LustreError: 366089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11331.060656] LustreError: 366089:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11333.007206] LustreError: 366070:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786590 with bad export cookie 15845391760925549478 [11333.014361] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11333.467747] Lustre: server umount lustre-MDT0001 complete [11345.710587] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11348.295782] Key type lgssc unregistered [11348.610972] LNet: 369620:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11348.621321] LNetError: 369620:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11348.650554] LNet: Removed LNI 192.168.204.125@tcp [11349.660701] Key type .llcrypt unregistered [11349.662332] Key type ._llcrypt unregistered [11380.408753] Key type ._llcrypt registered [11380.415370] Key type .llcrypt registered [11380.574804] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [11381.601237] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11381.676897] alg: No test for adler32 (adler32-zlib) [11382.797124] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [11383.069414] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [11384.808144] Key type lgssc registered [11385.828378] Lustre: Echo OBD driver; http://www.lustre.org/ [11393.246816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11393.282708] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11393.291150] LustreError: 371061:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11393.317328] LustreError: 371061:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11393.324250] LustreError: 371061:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11393.346942] LustreError: 371061:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11393.366270] LustreError: 371061:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11400.447327] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11400.457186] LustreError: 371460:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11400.493090] LustreError: 371460:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11400.507363] LustreError: 371460:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11400.517651] LustreError: 371460:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11400.536918] LustreError: 371460:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11420.405222] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11423.639620] Key type lgssc unregistered [11423.960773] LNet: 372463:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11423.974162] LNetError: 372463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11423.998051] LNet: Removed LNI 192.168.204.125@tcp [11425.153469] Key type .llcrypt unregistered [11425.156457] Key type ._llcrypt unregistered [11444.809070] Key type ._llcrypt registered [11444.811096] Key type .llcrypt registered [11444.918260] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [11461.478871] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [11463.263978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11463.290947] alg: No test for adler32 (adler32-zlib) [11464.493355] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [11464.847760] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [11466.543196] Key type lgssc registered [11467.634970] Lustre: Echo OBD driver; http://www.lustre.org/ [11502.466172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11502.480869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11503.729394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11503.766616] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11503.883881] Lustre: lustre-MDT0000: new disk, initializing [11503.989744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11504.002102] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11508.094623] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11518.663265] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11518.785308] Lustre: 375616:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11518.845841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11518.849354] Lustre: Skipped 1 previous similar message [11518.939680] Lustre: lustre-MDT0001: new disk, initializing [11519.050744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11519.109585] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11519.131212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11524.598484] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11535.045528] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11541.670922] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11549.436289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11549.695373] Lustre: lustre-OST0000: new disk, initializing [11549.699093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11549.705114] Lustre: 377583:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11549.823134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11555.572087] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11558.487444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11558.502246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11558.676553] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11565.015332] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11570.086338] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11570.319791] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11574.916902] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11575.181935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11579.370272] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11579.374970] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11579.390378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11579.396848] LustreError: Skipped 1 previous similar message [11579.411803] Lustre: Skipped 1 previous similar message [11583.355888] Lustre: server umount lustre-OST0000 complete [11589.601762] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11589.607568] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11589.627509] Lustre: Skipped 2 previous similar messages [11589.634499] Lustre: Skipped 1 previous similar message [11593.614296] Lustre: server umount lustre-MDT0000 complete [11597.782584] LustreError: 376410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788786854 with bad export cookie 13800711611004045391 [11597.799701] LustreError: 376410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11597.802356] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11598.246898] Lustre: server umount lustre-MDT0001 complete [11609.649464] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11611.390666] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 09:14:27 (1788786867) [11618.054674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11618.575695] LustreError: 380042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11618.599481] LustreError: 380042:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11618.659819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11622.453021] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11623.906072] LustreError: 380043:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11628.006558] LustreError: 380042:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11631.012794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11631.399485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11635.185928] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11642.835447] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11649.175751] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11656.704908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11657.060771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11663.980665] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11673.896955] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11681.221220] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11681.378504] Lustre: lustre-OST0001: new disk, initializing [11681.381357] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11681.386731] Lustre: 382877:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11681.515779] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11688.171719] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11689.090760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11689.107499] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11689.247619] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11696.688706] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11703.391458] Lustre: lustre-OST0001: Not available for connect from 192.168.204.25@tcp (stopping) [11704.840087] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11704.841776] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11706.436400] Lustre: server umount lustre-OST0001 complete [11708.469493] LustreError: 383298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11715.049572] LustreError: 381863:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11715.068239] LustreError: 381863:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11724.769096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11724.787747] LustreError: Skipped 1 previous similar message [11724.796792] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11724.806856] Lustre: Skipped 1 previous similar message [11724.815414] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11724.822493] Lustre: Skipped 2 previous similar messages [11725.290403] LustreError: 383298:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11725.315591] LustreError: 383298:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11729.519112] Lustre: server umount lustre-OST0000 complete [11733.988850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11734.000503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11734.010812] Lustre: Skipped 1 previous similar message [11734.014891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11734.018604] Lustre: Skipped 2 previous similar messages [11739.803406] Lustre: server umount lustre-MDT0000 complete [11744.149899] LustreError: 380024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787001 with bad export cookie 13800711611004046273 [11744.160058] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11750.539971] Lustre: server umount lustre-MDT0001 complete [11763.293990] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [11766.816562] Key type lgssc unregistered [11767.262751] LNet: 384868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11767.270735] LNetError: 384868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11767.304275] LNet: Removed LNI 192.168.204.125@tcp [11768.268153] Key type .llcrypt unregistered [11768.274690] Key type ._llcrypt unregistered [11826.243665] Key type ._llcrypt registered [11826.246031] Key type .llcrypt registered [11826.374291] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [11827.682841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11827.756065] alg: No test for adler32 (adler32-zlib) [11828.887975] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [11829.279562] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [11830.991245] Key type lgssc registered [11832.276819] Lustre: Echo OBD driver; http://www.lustre.org/ [11839.721707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11839.745118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11840.995491] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11841.014281] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11841.210800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11846.072784] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11856.705246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11856.873336] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11856.893656] Lustre: 387663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11857.144352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11862.822426] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11872.420284] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11878.110911] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11882.467435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11882.470641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11882.486277] Lustre: Skipped 1 previous similar message [11882.511793] Lustre: Skipped 1 previous similar message [11886.550769] Lustre: server umount lustre-MDT0000 complete [11890.212241] LustreError: 388406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787147 with bad export cookie 1015510348215306405 [11890.221198] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11890.224461] LustreError: 388406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11890.551267] Lustre: server umount lustre-MDT0001 complete [11900.315194] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 09:19:16 (1788787156) [11908.749263] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11909.265294] LustreError: 390177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11909.279388] LustreError: 390177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11909.349903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11914.723974] LustreError: 390176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11915.209591] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11919.843419] LustreError: 390177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11923.799080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11924.388430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11928.705577] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11936.178051] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11943.192596] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11950.888324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11951.111680] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11951.413589] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11958.498738] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [11968.566138] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11976.786787] Lustre: Permanently deactivating lustre-OST0000 [11976.806935] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11985.497766] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11987.100182] LustreError: 393163:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11987.120872] LustreError: 393163:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11987.127502] LustreError: 393163:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11990.772107] LustreError: 393213:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11990.779737] LustreError: 393213:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11999.521483] Lustre: server umount lustre-OST0000 complete [12002.445453] LustreError: 393463:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12002.456557] LustreError: 393463:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12006.375096] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12006.382207] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12006.391411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12007.393891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12007.396372] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12007.413697] Lustre: Skipped 1 previous similar message [12012.513925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12012.523633] Lustre: Skipped 1 previous similar message [12014.710418] Lustre: server umount lustre-MDT0000 complete [12017.633055] LustreError: 390177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12017.662516] LustreError: 390177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12018.844870] LustreError: 390908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787275 with bad export cookie 1015510348215306979 [12018.850764] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12018.867026] LustreError: 390908:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12028.597222] Lustre: server umount lustre-MDT0001 complete [12037.397501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12040.959896] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12043.517435] LustreError: 394622:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12049.586731] Lustre: 394961:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.125@tcp [12052.029737] Lustre: server umount MGS complete [12065.795603] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12066.205586] LustreError: 395686:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12066.225698] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12066.232584] Lustre: Skipped 1 previous similar message [12066.311192] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12071.556727] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12076.512946] LustreError: 395687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12076.554662] LustreError: 395687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12083.347418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12083.779554] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12083.895665] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12089.602519] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12098.626792] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12105.443617] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12108.078200] Lustre: Permanently reactivating lustre-OST0000 [12108.117418] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12108.128602] Lustre: Skipped 2 previous similar messages [12136.927430] LustreError: 397470:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12145.667393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12146.038358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12153.393723] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12161.983815] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12171.744205] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12171.760577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12171.800351] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12172.773853] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12174.080323] Lustre: server umount lustre-OST0000 complete [12177.888456] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12177.900902] Lustre: Skipped 2 previous similar messages [12183.020152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12183.024102] Lustre: Skipped 5 previous similar messages [12192.223406] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12192.553795] Lustre: server umount lustre-MDT0000 complete [12193.249821] LustreError: 395683:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12193.278852] LustreError: 395683:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12196.302574] LustreError: 395667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787453 with bad export cookie 1015510348215308036 [12196.310710] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12196.328947] LustreError: 395667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12196.786677] Lustre: server umount lustre-MDT0001 complete [12207.534295] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12210.639749] Key type lgssc unregistered [12211.040681] LNet: 400054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12211.049053] LNetError: 400054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12211.070712] LNet: Removed LNI 192.168.204.125@tcp [12212.046742] Key type .llcrypt unregistered [12212.048952] Key type ._llcrypt unregistered [12231.907597] Key type ._llcrypt registered [12231.910078] Key type .llcrypt registered [12232.073796] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [12248.214435] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [12249.199792] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12249.295423] alg: No test for adler32 (adler32-zlib) [12250.323595] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [12250.632615] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [12252.367950] Key type lgssc registered [12253.644594] Lustre: Echo OBD driver; http://www.lustre.org/ [12292.463257] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12292.477312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12293.697852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12293.767622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12293.836505] Lustre: lustre-MDT0000: new disk, initializing [12293.934309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12293.954767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12298.066950] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12307.854472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12307.970866] Lustre: 403208:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12308.029965] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12308.034396] Lustre: Skipped 1 previous similar message [12308.081659] Lustre: lustre-MDT0001: new disk, initializing [12308.167562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12308.201895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12308.211148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12312.915482] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12323.506726] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12330.431536] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12336.731045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12336.903067] Lustre: lustre-OST0000: new disk, initializing [12336.907974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12336.912034] Lustre: 405174:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12336.975471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12343.085759] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12346.423946] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12346.451889] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12346.598060] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12354.694202] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12359.554259] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12359.721951] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12364.865920] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12365.080759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12372.453287] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12372.456906] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12372.480163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12372.486072] Lustre: Skipped 1 previous similar message [12373.921601] Lustre: server umount lustre-OST0000 complete [12380.130203] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12380.137228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12380.151229] Lustre: Skipped 1 previous similar message [12380.162098] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12382.694177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12382.703073] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12384.526928] Lustre: server umount lustre-MDT0000 complete [12388.623551] LustreError: 403200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787645 with bad export cookie 17726216078116758930 [12388.630469] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12388.637865] LustreError: 403200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12389.067888] Lustre: server umount lustre-MDT0001 complete [12399.664647] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12401.630306] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 09:27:37 (1788787657) [12410.123709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12411.143678] LustreError: 407637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12411.201556] LustreError: 407637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12411.304201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12416.487724] LustreError: 407638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12418.040715] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12421.601210] LustreError: 407637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12426.739430] LustreError: 407638:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12429.544785] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12430.044047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12435.773815] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12445.020976] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12451.333536] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12458.997708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12459.313631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12466.253471] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12474.909862] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12477.189729] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12484.497916] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12492.263869] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12492.271755] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12492.285864] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12492.285871] Lustre: Skipped 1 previous similar message [12496.644952] Lustre: server umount lustre-OST0000 complete [12501.472820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12501.492264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12501.518359] Lustre: Skipped 1 previous similar message [12501.525543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12501.531730] Lustre: Skipped 1 previous similar message [12512.737291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12512.747194] Lustre: Skipped 5 previous similar messages [12515.295201] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12515.513384] Lustre: server umount lustre-MDT0000 complete [12517.865914] LustreError: 407637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12517.897557] LustreError: 407637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12520.645220] LustreError: 407619:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787777 with bad export cookie 17726216078116759812 [12520.660136] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12521.203032] Lustre: server umount lustre-MDT0001 complete [12533.156517] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12535.935158] Key type lgssc unregistered [12536.179687] LNet: 411363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12536.188199] LNetError: 411363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12536.220263] LNet: Removed LNI 192.168.204.125@tcp [12537.262499] Key type .llcrypt unregistered [12537.265153] Key type ._llcrypt unregistered [12563.182877] Key type ._llcrypt registered [12563.185601] Key type .llcrypt registered [12563.284377] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12563.551490] Key type .llcrypt unregistered [12563.555260] Key type ._llcrypt unregistered [12581.491489] Key type ._llcrypt registered [12581.496928] Key type .llcrypt registered [12581.650579] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [12582.606451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12582.727818] alg: No test for adler32 (adler32-zlib) [12583.828532] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [12584.036629] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [12585.743301] Key type lgssc registered [12587.122266] Lustre: Echo OBD driver; http://www.lustre.org/ [12596.186058] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12596.236496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12597.788507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12603.025439] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12607.975625] LustreError: 413570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12607.992715] LustreError: 413570:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12613.088607] LustreError: 413571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12614.123369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12614.592204] LustreError: 413570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12614.632894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12619.755361] LustreError: 413567:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12619.779418] LustreError: 413567:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12622.847433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12622.927526] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12627.951438] LustreError: 413571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12627.983113] LustreError: 413571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12632.506107] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12638.187468] LustreError: 413566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12638.235927] LustreError: 413566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12641.678038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12642.167621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12647.493927] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12657.637697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12657.639263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12657.645207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12658.691323] Lustre: server umount lustre-OST0000 complete [12662.753986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12662.755260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12662.763109] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12662.767429] Lustre: Skipped 2 previous similar messages [12662.780284] Lustre: Skipped 2 previous similar messages [12664.880581] Lustre: server umount lustre-MDT0000 complete [12667.872974] LustreError: 413571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12667.899078] LustreError: 413571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12669.789578] LustreError: 413550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788787926 with bad export cookie 12166011539317352300 [12669.815348] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12669.818329] LustreError: 413550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12670.353101] Lustre: server umount lustre-MDT0001 complete [12682.918177] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12686.486966] Key type lgssc unregistered [12686.748688] LNet: 416718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12686.759770] LNetError: 416718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12686.779526] LNet: Removed LNI 192.168.204.125@tcp [12687.884845] Key type .llcrypt unregistered [12687.887332] Key type ._llcrypt unregistered [12722.227840] Key type ._llcrypt registered [12722.235863] Key type .llcrypt registered [12722.480284] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [12723.432643] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12723.470681] alg: No test for adler32 (adler32-zlib) [12724.491440] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [12724.648771] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [12726.271188] Key type lgssc registered [12727.221656] Lustre: Echo OBD driver; http://www.lustre.org/ [12734.902493] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12734.928627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12750.495878] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12755.616534] LustreError: 418107:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.125@tcp: failed processing log, type 4: rc = -110 [12782.239220] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12788.837905] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12796.398382] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12804.704326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12805.315324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12805.417680] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12809.864411] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12810.729209] LustreError: 418132:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12810.781343] LustreError: 418132:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12815.844770] LustreError: 418133:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.897805] LustreError: 419143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12817.914761] LustreError: 419143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12819.413705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12819.859693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12825.161906] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12831.923106] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12838.447686] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12850.666386] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12850.667094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12850.671136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12850.690513] Lustre: Skipped 1 previous similar message [12853.567580] Lustre: server umount lustre-OST0000 complete [12860.896905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12860.898075] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12860.927333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12860.940436] Lustre: Skipped 2 previous similar messages [12863.642217] Lustre: server umount lustre-MDT0000 complete [12866.023424] LustreError: 420393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12866.047721] LustreError: 420393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12867.345605] LustreError: 420725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788788124 with bad export cookie 10714332631552819524 [12867.347561] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12867.361613] LustreError: 420725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12867.737802] Lustre: server umount lustre-MDT0001 complete [12878.677797] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [12881.869184] Key type lgssc unregistered [12882.174603] LNet: 421679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12882.181556] LNetError: 421679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12882.214740] LNet: Removed LNI 192.168.204.125@tcp [12883.170561] Key type .llcrypt unregistered [12883.174919] Key type ._llcrypt unregistered [12907.594695] Key type ._llcrypt registered [12907.596517] Key type .llcrypt registered [12907.689788] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [12908.344364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12908.447994] alg: No test for adler32 (adler32-zlib) [12909.588774] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [12909.862366] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [12911.495894] Key type lgssc registered [12912.641491] Lustre: Echo OBD driver; http://www.lustre.org/ [12921.963315] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12921.991750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12923.573575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12928.921889] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12933.094109] LustreError: 423094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12933.124917] LustreError: 423094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12938.212500] LustreError: 423093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12938.802115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12939.181854] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12944.754867] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12955.555528] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12962.675222] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12969.983565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12970.517453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12977.205455] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [12985.617438] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12993.138556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13000.678021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13000.689242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13000.704756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13002.853609] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.25@tcp (stopping) [13003.748278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13003.771519] Lustre: Skipped 1 previous similar message [13006.943449] Lustre: server umount lustre-MDT0000 complete [13007.941805] LustreError: 423088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13010.961426] LustreError: 423074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.25@tcp arrived at 1788788267 with bad export cookie 18006559993325690001 [13011.010215] LustreError: 423074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13013.931708] LustreError: 425811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788788270 with bad export cookie 18006559993325689336 [13013.959103] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13013.990093] LustreError: 423093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13013.993264] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13014.022827] LustreError: 423093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13014.065685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13014.082042] Lustre: Skipped 3 previous similar messages [13019.108617] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13019.123317] LustreError: 423090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13019.173531] LustreError: 423090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13020.418367] Lustre: server umount lustre-OST0000 complete [13028.320851] LustreError: 423821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13028.348014] LustreError: 423821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13030.360276] Lustre: server umount lustre-MDT0001 complete [13041.880474] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13044.764638] Key type lgssc unregistered [13045.016561] LNet: 426801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13045.022416] LNetError: 426801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13045.048266] LNet: Removed LNI 192.168.204.125@tcp [13045.767773] Key type .llcrypt unregistered [13045.770831] Key type ._llcrypt unregistered [13074.644302] Key type ._llcrypt registered [13074.646803] Key type .llcrypt registered [13074.754838] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13075.986656] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13076.087866] alg: No test for adler32 (adler32-zlib) [13077.152362] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [13077.534041] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [13079.311180] Key type lgssc registered [13080.521980] Lustre: Echo OBD driver; http://www.lustre.org/ [13089.191937] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13089.210516] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13090.671077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13094.685468] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13100.002626] LustreError: 428215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13100.036455] LustreError: 428215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13103.516773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13103.852792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13108.954174] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13116.602375] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13122.369935] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13129.063989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13129.390420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13135.556569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13136.097051] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13143.676991] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13150.183051] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13150.201213] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13150.225348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13151.800261] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.25@tcp (stopping) [13154.277708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13154.288895] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13154.318587] Lustre: Skipped 1 previous similar message [13155.794395] Lustre: server umount lustre-MDT0001 complete [13156.938408] LustreError: 429457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.25@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13159.393947] LustreError: 428216:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13159.440845] LustreError: 428216:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13164.532579] LustreError: 428215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13164.536084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13164.578600] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13168.855399] Lustre: server umount lustre-OST0000 complete [13169.653371] LustreError: 430337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13172.922570] Lustre: server umount lustre-MDT0000 complete [13188.363864] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13191.454408] Key type lgssc unregistered [13191.738974] LNet: 431900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13191.751067] LNetError: 431900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13191.766822] LNet: Removed LNI 192.168.204.125@tcp [13192.669959] Key type .llcrypt unregistered [13192.671701] Key type ._llcrypt unregistered [13224.508568] Key type ._llcrypt registered [13224.511127] Key type .llcrypt registered [13224.664933] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [13242.422674] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13243.463268] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13243.556046] alg: No test for adler32 (adler32-zlib) [13244.672049] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [13245.109665] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [13246.887210] Key type lgssc registered [13248.067153] Lustre: Echo OBD driver; http://www.lustre.org/ [13290.136314] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13308.123906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13308.154872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13309.460553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13309.504625] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13309.617681] Lustre: lustre-MDT0000: new disk, initializing [13309.786409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13309.819347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13314.737867] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13331.758301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13332.016355] Lustre: lustre-OST0000: new disk, initializing [13332.024631] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13332.030714] Lustre: Skipped 1 previous similar message [13332.040565] Lustre: 436935:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13332.126494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13339.192682] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13342.298400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13342.326661] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13342.367075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13349.360113] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13371.025602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13371.242368] Lustre: 435914:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13371.386274] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13371.542406] Lustre: lustre-MDT0001: new disk, initializing [13371.697836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13371.761134] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13371.780775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13377.179656] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13387.927649] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13388.253390] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13394.742606] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13395.139172] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13401.184684] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.25@tcp (stopping) [13402.080396] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13402.091818] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13402.094173] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13402.121882] Lustre: Skipped 1 previous similar message [13406.265842] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.25@tcp (stopping) [13406.273865] Lustre: Skipped 1 previous similar message [13411.384822] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.25@tcp (stopping) [13411.389742] Lustre: Skipped 2 previous similar messages [13415.391277] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13415.722170] Lustre: server umount lustre-MDT0001 complete [13422.564317] LustreError: 437217:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13422.587333] LustreError: 437217:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13427.685409] LustreError: 439508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13427.717595] LustreError: 439508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13432.801386] LustreError: 435928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13432.834673] LustreError: 435928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13437.926501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13437.938780] LustreError: 438472:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13437.951248] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13437.983142] Lustre: Skipped 2 previous similar messages [13441.421450] Lustre: server umount lustre-OST0000 complete [13443.044083] LustreError: 435928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13452.456791] Lustre: server umount lustre-MDT0000 complete [13472.276062] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13475.530704] Key type lgssc unregistered [13475.749597] LNet: 440690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13475.757451] LNetError: 440690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13475.775292] LNet: Removed LNI 192.168.204.125@tcp [13476.466097] Key type .llcrypt unregistered [13476.471749] Key type ._llcrypt unregistered [13522.994435] Key type ._llcrypt registered [13523.004614] Key type .llcrypt registered [13523.114432] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [13523.472506] Key type .llcrypt unregistered [13523.477393] Key type ._llcrypt unregistered [13539.478958] Key type ._llcrypt registered [13539.487824] Key type .llcrypt registered [13539.642878] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [13540.755526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13540.934471] alg: No test for adler32 (adler32-zlib) [13541.988100] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [13542.208494] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [13543.919315] Key type lgssc registered [13545.044118] Lustre: Echo OBD driver; http://www.lustre.org/ [13572.276888] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13572.311670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13573.575156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13573.615537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13573.713733] Lustre: lustre-MDT0000: new disk, initializing [13573.828966] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13573.841150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13577.938735] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13588.801120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13588.954758] Lustre: 444060:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13589.004199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13589.014186] Lustre: Skipped 1 previous similar message [13589.089582] Lustre: lustre-MDT0001: new disk, initializing [13589.187685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13589.218248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13589.232050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13594.763337] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13605.360806] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13611.679204] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13619.395753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13619.815686] Lustre: lustre-OST0000: new disk, initializing [13619.819653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13619.833992] Lustre: 446030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13619.951166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13626.925516] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13628.527945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13628.545199] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13628.680177] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13637.687324] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13645.796807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13645.812956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13645.839888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13649.381169] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13649.383776] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13649.394820] Lustre: Skipped 1 previous similar message [13649.423596] Lustre: Skipped 1 previous similar message [13652.244282] Lustre: server umount lustre-MDT0000 complete [13657.188798] LustreError: 444052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788788914 with bad export cookie 8024415600284685751 [13657.201812] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13657.206710] LustreError: 444052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13657.876273] Lustre: server umount lustre-MDT0001 complete [13672.404625] Lustre: server umount lustre-OST0000 complete [13683.940075] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 09:48:59 (1788788939) [13691.925085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13692.684218] LustreError: 448058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13692.719530] LustreError: 448058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13692.806654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13697.910220] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13698.016847] LustreError: 448059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13702.116593] LustreError: 448058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13707.246366] LustreError: 448059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13707.298844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13707.884038] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13713.902356] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13722.546901] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13730.380962] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13751.927327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13752.291449] Lustre: Found index 0 for lustre-OST0000, updating log [13752.730425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13761.829644] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [13764.032462] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13770.535482] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13778.916162] Lustre: server umount lustre-OST0000 complete [13796.191865] Lustre: 442591:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788789036/real 1788789036] req@0000000044e47f93 x1875681003774592/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788789052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13796.213522] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13796.231040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13796.232934] Lustre: Skipped 1 previous similar message [13800.353460] Lustre: 442592:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788789041/real 1788789041] req@00000000c1807225 x1875681003775744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788789057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13800.384947] Lustre: 442592:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13801.440989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13801.445448] Lustre: Skipped 1 previous similar message [13805.407204] Lustre: 442593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788789046/real 1788789046] req@000000005c561cd0 x1875681003777024/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788789062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13805.686463] Lustre: server umount lustre-MDT0000 complete [13806.567315] LustreError: 448059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13806.612319] LustreError: 448059:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13810.611921] Lustre: 442593:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788789051/real 1788789051] req@00000000cf0e5430 x1875681003778048/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788789067 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13812.966487] LustreError: 448040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788789069 with bad export cookie 8024415600284687151 [13812.984397] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13812.997161] LustreError: 448040:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13813.054686] LustreError: 442594:0:(import.c:717:ptlrpc_connect_import_locked()) already connecting [13813.865231] Lustre: server umount lustre-MDT0001 complete [13819.751265] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp [13828.040239] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 09:51:23 (1788789083) [14454.147339] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14455.155362] LustreError: 454486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14455.182883] LustreError: 454486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14455.291213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14460.051490] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14460.390989] LustreError: 454487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14465.520477] LustreError: 454486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14469.688898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14470.034522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14473.961164] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14482.806391] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14488.814639] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14508.790917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14509.200660] Lustre: Found index 0 for lustre-OST0000, updating log [14509.451769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14510.949720] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp [14516.367145] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14523.412565] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14525.500365] Lustre: server umount lustre-OST0000 complete [14529.834172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14529.842599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14529.850564] Lustre: Skipped 3 previous similar messages [14529.870120] Lustre: Skipped 1 previous similar message [14530.915685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14530.919446] Lustre: Skipped 1 previous similar message [14531.943136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14534.052197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14534.067995] Lustre: Skipped 4 previous similar messages [14538.218325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14538.230141] Lustre: Skipped 7 previous similar messages [14543.327244] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14543.477240] Lustre: server umount lustre-MDT0000 complete [14544.364370] LustreError: 454481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14544.398878] LustreError: 454481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14549.118272] LustreError: 454468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788789806 with bad export cookie 8024415600284688236 [14549.125577] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14549.132873] LustreError: 454468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14549.930342] Lustre: server umount lustre-MDT0001 complete [14570.341189] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp [14581.976432] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [14600.505947] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [14642.328373] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 10:04:57 (1788789897) [14665.955150] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [14669.559818] Key type lgssc unregistered [14669.902297] LNet: 461129:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14669.918922] LNetError: 461129:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14771.303981] LNet: Removed LNI 192.168.204.125@tcp [14772.282855] Key type .llcrypt unregistered [14772.284792] Key type ._llcrypt unregistered [14790.728456] Key type ._llcrypt registered [14790.731136] Key type .llcrypt registered [14790.855872] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [14791.938648] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14792.171915] alg: No test for adler32 (adler32-zlib) [14793.351464] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [14793.706350] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [14795.440944] Key type lgssc registered [14796.801555] Lustre: Echo OBD driver; http://www.lustre.org/ [14801.928864] LNet: Added LNI 192.168.204.125@tcp100 [8/256/0/180] [14803.003887] LNet: Added LNI 192.168.204.125@tcp101 [8/256/0/180] [14803.903271] LNet: Added LNI 192.168.204.125@tcp102 [8/256/0/180] [14804.952069] LNet: Added LNI 192.168.204.125@tcp103 [8/256/0/180] [14805.919437] LNet: Added LNI 192.168.204.125@tcp104 [8/256/0/180] [14806.965377] LNet: Added LNI 192.168.204.125@tcp105 [8/256/0/180] [14807.814264] LNet: Added LNI 192.168.204.125@tcp106 [8/256/0/180] [14808.672842] LNet: Added LNI 192.168.204.125@tcp107 [8/256/0/180] [14809.627317] LNet: Added LNI 192.168.204.125@tcp108 [8/256/0/180] [14810.640731] LNet: Added LNI 192.168.204.125@tcp109 [8/256/0/180] [14811.557055] LNet: Added LNI 192.168.204.125@tcp110 [8/256/0/180] [14812.533898] LNet: Added LNI 192.168.204.125@tcp111 [8/256/0/180] [14813.592510] LNet: Added LNI 192.168.204.125@tcp112 [8/256/0/180] [14814.509470] LNet: Added LNI 192.168.204.125@tcp113 [8/256/0/180] [14815.487946] LNet: Added LNI 192.168.204.125@tcp114 [8/256/0/180] [14816.371533] LNet: Added LNI 192.168.204.125@tcp115 [8/256/0/180] [14817.299434] LNet: Added LNI 192.168.204.125@tcp116 [8/256/0/180] [14818.272767] LNet: Added LNI 192.168.204.125@tcp117 [8/256/0/180] [14819.274791] LNet: Added LNI 192.168.204.125@tcp118 [8/256/0/180] [14820.329591] LNet: Added LNI 192.168.204.125@tcp119 [8/256/0/180] [14821.341669] LNet: Added LNI 192.168.204.125@tcp120 [8/256/0/180] [14822.300452] LNet: Added LNI 192.168.204.125@tcp121 [8/256/0/180] [14823.377034] LNet: Added LNI 192.168.204.125@tcp122 [8/256/0/180] [14824.519717] LNet: Added LNI 192.168.204.125@tcp123 [8/256/0/180] [14825.424884] LNet: Added LNI 192.168.204.125@tcp124 [8/256/0/180] [14826.470406] LNet: Added LNI 192.168.204.125@tcp125 [8/256/0/180] [14827.514478] LNet: Added LNI 192.168.204.125@tcp126 [8/256/0/180] [14828.659633] LNet: Added LNI 192.168.204.125@tcp127 [8/256/0/180] [14829.819373] LNet: Added LNI 192.168.204.125@tcp128 [8/256/0/180] [14830.918655] LNet: Added LNI 192.168.204.125@tcp129 [8/256/0/180] [14831.965521] LNet: Added LNI 192.168.204.125@tcp130 [8/256/0/180] [14832.923171] LNet: Added LNI 192.168.204.125@tcp131 [8/256/0/180] [14833.891530] LNet: Added LNI 192.168.204.125@tcp132 [8/256/0/180] [14834.789348] LNet: Added LNI 192.168.204.125@tcp133 [8/256/0/180] [14835.689389] LNet: Added LNI 192.168.204.125@tcp134 [8/256/0/180] [14836.430869] LNet: Added LNI 192.168.204.125@tcp135 [8/256/0/180] [14844.201499] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14844.257491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14845.439387] Lustre: 464158:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [14845.518014] LNetError: 464182:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [14845.559982] LNetError: Protocol error connecting to 192.168.204.125@tcp121 at host 192.168.204.125:988: is it running a compatible version of Lustre? [14845.573965] LNetError: Skipped 2 previous similar messages [14845.677242] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14845.712233] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14845.831682] Lustre: lustre-MDT0000: new disk, initializing [14845.915339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14845.969777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14845.991306] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp121 [14846.632642] LNetError: 464186:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [14846.660387] LNetError: 464186:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [14846.674387] LNetError: Protocol error connecting to 192.168.204.125@tcp106 at host 192.168.204.125:988: is it running a compatible version of Lustre? [14849.058080] LNetError: 464181:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [14849.089691] LNetError: 464181:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [14849.102065] LNetError: Protocol error connecting to 192.168.204.125@tcp121 at host 192.168.204.125:988: is it running a compatible version of Lustre? [14849.118846] LNetError: Skipped 1 previous similar message [14851.077614] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14853.152812] LNetError: 461615:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [14853.162982] LNetError: Protocol error connecting to 192.168.204.125@tcp121 at host 192.168.204.125:988: is it running a compatible version of Lustre? [14861.352578] LNetError: 464180:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [14861.374844] LNetError: Protocol error connecting to 192.168.204.125@tcp121 at host 192.168.204.125:988: is it running a compatible version of Lustre? [14863.018724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14863.184579] Lustre: 464175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14863.249468] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14863.252529] Lustre: Skipped 1 previous similar message [14863.321264] Lustre: lustre-MDT0001: new disk, initializing [14863.352237] Lustre: 464977:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [14863.362104] Lustre: 464977:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14863.480498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14863.526987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14863.534672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14868.165557] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14871.052852] Lustre: 465340:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [14871.059427] Lustre: 465340:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14876.644024] LNetError: 461616:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [14876.652526] LNetError: Protocol error connecting to 192.168.204.125@tcp128 at host 192.168.204.125:988: is it running a compatible version of Lustre? [14877.939691] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14883.703508] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14889.872267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14890.153727] Lustre: lustre-OST0000: new disk, initializing [14890.159316] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14890.168617] Lustre: 466149:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14890.270519] Lustre: 466154:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [14890.275963] Lustre: 466154:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14890.337236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14894.053375] LNetError: 461617:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [14894.063109] LNetError: 461617:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14894.071690] LNetError: Protocol error connecting to 192.168.204.125@tcp129 at host 192.168.204.125:988: is it running a compatible version of Lustre? [14894.088328] LNetError: Skipped 4 previous similar messages [14896.303901] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [14896.682745] Lustre: 466517:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.125@tcp: can't add NID 192.168.204.125@tcp130: rc = -75 [14896.710141] Lustre: 466517:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14896.762324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14896.770887] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14896.846992] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14905.375207] LNet: 1 local NIs in recovery (showing 1): 192.168.204.125@tcp121 [14906.734229] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14917.608680] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14917.610368] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14917.615474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14917.621369] LustreError: Skipped 1 previous similar message [14917.641616] Lustre: Skipped 1 previous similar message [14922.732436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14922.743327] Lustre: Skipped 2 previous similar messages [14927.846437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14927.857528] Lustre: Skipped 1 previous similar message [14928.351139] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14928.468874] Lustre: server umount lustre-OST0000 complete [14932.960780] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14932.966272] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14932.995367] Lustre: Skipped 1 previous similar message [14933.021719] Lustre: Skipped 1 previous similar message [14933.985849] LNetError: 464187:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.125 [14934.013331] LNetError: 464187:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14934.024260] LNetError: Protocol error connecting to 192.168.204.125@tcp106 at host 192.168.204.125:988: is it running a compatible version of Lustre? [14934.032253] LNetError: Skipped 7 previous similar messages [14938.081786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14938.091264] Lustre: Skipped 1 previous similar message [14938.858207] Lustre: server umount lustre-MDT0000 complete [14943.658751] LustreError: 464166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788790200 with bad export cookie 6790831328606100297 [14943.666307] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14943.676659] LustreError: 464166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14944.204427] Lustre: server umount lustre-MDT0001 complete [14955.687218] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [14958.254737] Key type lgssc unregistered [14958.612907] LNet: 468062:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14958.619360] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14958.623629] LNet: Removed LNI 192.168.204.125@tcp [14959.647836] LNet: Removed LNI 192.168.204.125@tcp100 [14959.659149] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14959.679689] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14959.707768] LNet: Removed LNI 192.168.204.125@tcp101 [14960.735637] LNet: Removed LNI 192.168.204.125@tcp102 [14960.743558] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14960.754910] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14960.773103] LNet: Removed LNI 192.168.204.125@tcp103 [14961.823745] LNet: Removed LNI 192.168.204.125@tcp104 [14962.847233] LNet: Removed LNI 192.168.204.125@tcp105 [14962.850837] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14962.856308] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14962.860559] LNet: Removed LNI 192.168.204.125@tcp106 [14963.871216] LNet: Removed LNI 192.168.204.125@tcp107 [14964.895421] LNet: Removed LNI 192.168.204.125@tcp108 [14965.919374] LNet: Removed LNI 192.168.204.125@tcp109 [14967.007147] LNet: Removed LNI 192.168.204.125@tcp110 [14967.013082] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14967.024786] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14967.038870] LNet: Removed LNI 192.168.204.125@tcp111 [14968.095244] LNet: Removed LNI 192.168.204.125@tcp112 [14969.119405] LNet: Removed LNI 192.168.204.125@tcp113 [14970.143138] LNet: Removed LNI 192.168.204.125@tcp114 [14971.170088] LNet: Removed LNI 192.168.204.125@tcp115 [14972.191460] LNet: Removed LNI 192.168.204.125@tcp116 [14973.215602] LNet: Removed LNI 192.168.204.125@tcp117 [14974.239219] LNet: Removed LNI 192.168.204.125@tcp118 [14975.263189] LNet: Removed LNI 192.168.204.125@tcp119 [14975.267749] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14975.279513] LNetError: 468062:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14975.296215] LNet: Removed LNI 192.168.204.125@tcp120 [14975.299570] LNet: Removed LNI 192.168.204.125@tcp121 [14976.351431] LNet: Removed LNI 192.168.204.125@tcp122 [14977.375208] LNet: Removed LNI 192.168.204.125@tcp123 [14978.400074] LNet: Removed LNI 192.168.204.125@tcp124 [14979.423904] LNet: Removed LNI 192.168.204.125@tcp125 [14980.448644] LNet: Removed LNI 192.168.204.125@tcp126 [14981.471174] LNet: Removed LNI 192.168.204.125@tcp127 [14982.496170] LNet: Removed LNI 192.168.204.125@tcp128 [14983.519304] LNet: Removed LNI 192.168.204.125@tcp129 [14983.521567] LNet: Removed LNI 192.168.204.125@tcp130 [14983.531605] LNet: Removed LNI 192.168.204.125@tcp131 [14983.539077] LNet: Removed LNI 192.168.204.125@tcp132 [14983.544230] LNet: Removed LNI 192.168.204.125@tcp133 [14983.549584] LNet: Removed LNI 192.168.204.125@tcp134 [14983.563562] LNet: Removed LNI 192.168.204.125@tcp135 [14984.257908] Key type .llcrypt unregistered [14984.264171] Key type ._llcrypt unregistered [15012.643796] Key type ._llcrypt registered [15012.647498] Key type .llcrypt registered [15012.812216] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [15028.684221] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15029.254115] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15029.268982] alg: No test for adler32 (adler32-zlib) [15030.231859] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [15030.444625] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [15032.079184] Key type lgssc registered [15032.999388] Lustre: Echo OBD driver; http://www.lustre.org/ [15067.980700] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 10:12:03 (1788790323) [15088.719475] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [15091.864817] Key type lgssc unregistered [15092.232928] LNet: 472147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15092.245497] LNetError: 472147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15092.270990] LNet: Removed LNI 192.168.204.125@tcp [15093.124532] Key type .llcrypt unregistered [15093.126309] Key type ._llcrypt unregistered [15114.401610] Key type ._llcrypt registered [15114.403698] Key type .llcrypt registered [15114.511908] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15115.536264] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15115.702740] alg: No test for adler32 (adler32-zlib) [15116.880180] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [15117.300061] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [15119.023225] Key type lgssc registered [15120.581078] Lustre: Echo OBD driver; http://www.lustre.org/ [15130.458601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15130.489345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15131.788773] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15131.830924] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15131.958937] Lustre: lustre-MDT0000: new disk, initializing [15132.107970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15132.132187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15136.859882] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15148.321927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15148.497242] Lustre: 473355:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15148.567849] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15148.573097] Lustre: Skipped 1 previous similar message [15148.640475] Lustre: lustre-MDT0001: new disk, initializing [15148.766972] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15148.868174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15148.892571] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15153.369290] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15163.680432] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15170.986397] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15178.596203] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15178.851912] Lustre: lustre-OST0000: new disk, initializing [15178.855845] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15178.861208] Lustre: 475327:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15178.933885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15184.954065] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15189.081414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15189.088068] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15189.296944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15196.355827] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15202.445302] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing lnet_if_list [15209.421541] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15209.481776] LNet: Added LNI 192.168.204.125@tcp10 [8/256/0/180] [15217.100818] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15217.135170] LNet: Added LNI 192.168.204.125@tcp11 [8/256/0/180] [15221.956030] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15221.992794] LNet: Added LNI 192.168.204.125@tcp12 [8/256/0/180] [15227.582477] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15227.619623] LNet: Added LNI 192.168.204.125@tcp13 [8/256/0/180] [15233.091522] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15233.134611] LNet: Added LNI 192.168.204.125@tcp14 [8/256/0/180] [15237.960217] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15237.993564] LNet: Added LNI 192.168.204.125@tcp15 [8/256/0/180] [15242.089477] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15242.109764] LNet: Added LNI 192.168.204.125@tcp16 [8/256/0/180] [15246.650738] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15246.697449] LNet: Added LNI 192.168.204.125@tcp17 [8/256/0/180] [15251.269105] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15251.305135] LNet: Added LNI 192.168.204.125@tcp18 [8/256/0/180] [15257.364842] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15257.400760] LNet: Added LNI 192.168.204.125@tcp19 [8/256/0/180] [15264.193891] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15264.238564] LNet: Added LNI 192.168.204.125@tcp20 [8/256/0/180] [15271.279839] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15271.308235] LNet: Added LNI 192.168.204.125@tcp21 [8/256/0/180] [15277.473964] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15277.511244] LNet: Added LNI 192.168.204.125@tcp22 [8/256/0/180] [15283.730142] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15283.769262] LNet: Added LNI 192.168.204.125@tcp23 [8/256/0/180] [15290.774262] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15290.815834] LNet: Added LNI 192.168.204.125@tcp24 [8/256/0/180] [15296.062948] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15296.097944] LNet: Added LNI 192.168.204.125@tcp25 [8/256/0/180] [15302.326620] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15302.382813] LNet: Added LNI 192.168.204.125@tcp26 [8/256/0/180] [15310.443670] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15310.467656] LNet: Added LNI 192.168.204.125@tcp27 [8/256/0/180] [15319.493369] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15319.529787] LNet: Added LNI 192.168.204.125@tcp28 [8/256/0/180] [15327.802267] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15327.830332] LNet: Added LNI 192.168.204.125@tcp29 [8/256/0/180] [15337.108854] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15337.135694] LNet: Added LNI 192.168.204.125@tcp30 [8/256/0/180] [15344.274967] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15344.304044] LNet: Added LNI 192.168.204.125@tcp31 [8/256/0/180] [15351.374897] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15351.400514] LNet: Added LNI 192.168.204.125@tcp32 [8/256/0/180] [15358.974331] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15358.999823] LNet: Added LNI 192.168.204.125@tcp33 [8/256/0/180] [15367.064848] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15367.094753] LNet: Added LNI 192.168.204.125@tcp34 [8/256/0/180] [15375.104030] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15375.160229] LNet: Added LNI 192.168.204.125@tcp35 [8/256/0/180] [15383.076250] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15383.134960] LNet: Added LNI 192.168.204.125@tcp36 [8/256/0/180] [15390.781963] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15390.804794] LNet: Added LNI 192.168.204.125@tcp37 [8/256/0/180] [15399.793502] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15399.840318] LNet: Added LNI 192.168.204.125@tcp38 [8/256/0/180] [15407.800892] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15407.837556] LNet: Added LNI 192.168.204.125@tcp39 [8/256/0/180] [15417.206658] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15417.225916] LNet: Added LNI 192.168.204.125@tcp40 [8/256/0/180] [15425.538235] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15425.578720] LNet: Added LNI 192.168.204.125@tcp41 [8/256/0/180] [15435.861819] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15435.906109] LNet: Added LNI 192.168.204.125@tcp42 [8/256/0/180] [15446.642433] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15446.674733] LNet: Added LNI 192.168.204.125@tcp43 [8/256/0/180] [15454.883874] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15454.937913] LNet: Added LNI 192.168.204.125@tcp44 [8/256/0/180] [15464.425682] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15464.446632] LNet: Added LNI 192.168.204.125@tcp45 [8/256/0/180] [15473.310877] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15473.336652] LNet: Added LNI 192.168.204.125@tcp46 [8/256/0/180] [15480.887834] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15480.902413] LNet: Added LNI 192.168.204.125@tcp47 [8/256/0/180] [15493.137040] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15493.185139] LNet: Added LNI 192.168.204.125@tcp48 [8/256/0/180] [15504.926915] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15505.008904] LNet: Added LNI 192.168.204.125@tcp49 [8/256/0/180] [15515.833560] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15515.883477] LNet: Added LNI 192.168.204.125@tcp50 [8/256/0/180] [15540.711737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15540.734285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15540.753274] Lustre: Skipped 2 previous similar messages [15540.780237] Lustre: Skipped 2 previous similar messages [15546.192049] Lustre: server umount lustre-MDT0000 complete [15550.946523] LustreError: 473364:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15550.957263] LustreError: 473364:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15552.011640] LustreError: 473349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788790809 with bad export cookie 1182301285937253022 [15552.012525] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15552.029984] LustreError: 473349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15552.383720] Lustre: server umount lustre-MDT0001 complete [15572.452115] Lustre: 472629:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788790813/real 1788790813] req@000000006c9018e7 x1875682655399040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788790829 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15572.480308] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15573.472158] Lustre: 472631:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788790814/real 1788790814] req@000000001a279dbb x1875682655399168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788790830 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15575.001121] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15577.570462] Lustre: 472631:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788790818/real 1788790818] req@00000000b5ed5e89 x1875682655399296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788790834 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15583.713317] Lustre: 472629:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788790824/real 1788790824] req@00000000d29fc574 x1875682655399680/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788790840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15583.743232] Lustre: 472629:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [15584.377891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15588.832332] Lustre: 472630:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788790829/real 1788790829] req@0000000081148ebd x1875682655399808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788790845 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15597.094435] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x106860c165c714ca [15597.110556] Lustre: MGC192.168.204.125@tcp: Connection restored to 0@lo (at 0@lo) [15597.380746] LustreError: 475320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15597.453311] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15602.341747] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15602.663117] LustreError: 475321:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15606.765713] LustreError: 475320:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15606.778340] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15611.486698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15611.782178] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15615.930430] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15618.031843] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15624.464795] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15631.814699] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15652.832677] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15652.854691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15652.874908] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15653.861644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15653.878224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15653.887500] Lustre: Skipped 1 previous similar message [15658.841776] Lustre: server umount lustre-OST0000 complete [15663.072755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15663.086820] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15663.100517] Lustre: Skipped 1 previous similar message [15663.108627] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15669.219441] LustreError: 499609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15669.437489] Lustre: server umount lustre-MDT0000 complete [15674.208231] LustreError: 473348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788790931 with bad export cookie 1182301285937255626 [15674.223152] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15674.232593] LustreError: 473348:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15674.811104] Lustre: server umount lustre-MDT0001 complete [15689.057593] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [15692.652683] Key type lgssc unregistered [15693.110773] LNet: 502407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15693.126120] LNetError: 502407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15693.135832] LNet: Removed LNI 192.168.204.125@tcp [15693.139623] LNet: Removed LNI 192.168.204.125@tcp10 [15693.143369] LNet: Removed LNI 192.168.204.125@tcp11 [15693.146838] LNet: Removed LNI 192.168.204.125@tcp12 [15693.149564] LNet: Removed LNI 192.168.204.125@tcp13 [15693.152548] LNet: Removed LNI 192.168.204.125@tcp14 [15693.155159] LNet: Removed LNI 192.168.204.125@tcp15 [15693.157636] LNet: Removed LNI 192.168.204.125@tcp16 [15693.160502] LNet: Removed LNI 192.168.204.125@tcp17 [15693.163874] LNet: Removed LNI 192.168.204.125@tcp18 [15693.169563] LNet: Removed LNI 192.168.204.125@tcp19 [15693.174421] LNet: Removed LNI 192.168.204.125@tcp20 [15693.177580] LNet: Removed LNI 192.168.204.125@tcp21 [15693.182622] LNet: Removed LNI 192.168.204.125@tcp22 [15693.187652] LNet: Removed LNI 192.168.204.125@tcp23 [15693.191718] LNet: Removed LNI 192.168.204.125@tcp24 [15693.194826] LNet: Removed LNI 192.168.204.125@tcp25 [15693.199396] LNet: Removed LNI 192.168.204.125@tcp26 [15693.202637] LNet: Removed LNI 192.168.204.125@tcp27 [15693.221190] LNet: Removed LNI 192.168.204.125@tcp28 [15693.232857] LNet: Removed LNI 192.168.204.125@tcp29 [15693.244904] LNet: Removed LNI 192.168.204.125@tcp30 [15693.250519] LNet: Removed LNI 192.168.204.125@tcp31 [15693.252851] LNet: Removed LNI 192.168.204.125@tcp32 [15693.257687] LNet: Removed LNI 192.168.204.125@tcp33 [15693.270817] LNet: Removed LNI 192.168.204.125@tcp34 [15693.275434] LNet: Removed LNI 192.168.204.125@tcp35 [15693.277411] LNet: Removed LNI 192.168.204.125@tcp36 [15693.282374] LNet: Removed LNI 192.168.204.125@tcp37 [15693.291378] LNet: Removed LNI 192.168.204.125@tcp38 [15693.295636] LNet: Removed LNI 192.168.204.125@tcp39 [15693.298929] LNet: Removed LNI 192.168.204.125@tcp40 [15693.302122] LNet: Removed LNI 192.168.204.125@tcp41 [15693.305676] LNet: Removed LNI 192.168.204.125@tcp42 [15693.315658] LNet: Removed LNI 192.168.204.125@tcp43 [15693.324687] LNet: Removed LNI 192.168.204.125@tcp44 [15693.333364] LNet: Removed LNI 192.168.204.125@tcp45 [15693.345814] LNet: Removed LNI 192.168.204.125@tcp46 [15693.357299] LNet: Removed LNI 192.168.204.125@tcp47 [15693.368141] LNet: Removed LNI 192.168.204.125@tcp48 [15693.369989] LNet: Removed LNI 192.168.204.125@tcp49 [15693.396323] LNet: Removed LNI 192.168.204.125@tcp50 [15694.198774] Key type .llcrypt unregistered [15694.200568] Key type ._llcrypt unregistered [15718.809949] Key type ._llcrypt registered [15718.811881] Key type .llcrypt registered [15718.937551] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [15735.377112] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15736.780825] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15736.890023] alg: No test for adler32 (adler32-zlib) [15738.053607] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [15738.284467] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [15739.975228] Key type lgssc registered [15741.477252] Lustre: Echo OBD driver; http://www.lustre.org/ [15781.946941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15781.980408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15783.179482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15783.201710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15783.257346] Lustre: lustre-MDT0000: new disk, initializing [15783.319719] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15783.343892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15788.584688] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15800.022290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15800.186525] Lustre: 505603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15800.259452] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15800.268434] Lustre: Skipped 1 previous similar message [15800.375973] Lustre: lustre-MDT0001: new disk, initializing [15800.463787] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15800.495485] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15800.532459] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15804.116830] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15813.922522] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15820.097151] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15828.565065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15828.703471] Lustre: *** cfs_fail_loc=90f, val=0*** [15828.763375] Lustre: lustre-OST0000: new disk, initializing [15828.766481] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15828.771840] Lustre: 507615:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15828.857455] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15837.628328] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15837.806386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15837.829769] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15838.169944] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15849.616464] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15858.663216] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15858.664846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15858.673050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15858.698228] LustreError: Skipped 1 previous similar message [15863.791040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15863.795980] Lustre: Skipped 1 previous similar message [15864.006256] Lustre: server umount lustre-OST0000 complete [15868.897280] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15868.910345] Lustre: Skipped 1 previous similar message [15868.916080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15868.922883] Lustre: Skipped 2 previous similar messages [15874.017076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15874.022795] Lustre: Skipped 1 previous similar message [15874.710916] Lustre: server umount lustre-MDT0000 complete [15878.440368] LustreError: 506396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791135 with bad export cookie 1057026381789755043 [15878.441601] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15878.456303] LustreError: 506396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15879.052892] Lustre: server umount lustre-MDT0001 complete [15892.690889] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [15896.145695] Key type lgssc unregistered [15896.512229] LNet: 509524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15896.533537] LNetError: 509524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15896.550309] LNet: Removed LNI 192.168.204.125@tcp [15897.500603] Key type .llcrypt unregistered [15897.507273] Key type ._llcrypt unregistered [15919.277136] Key type ._llcrypt registered [15919.280573] Key type .llcrypt registered [15919.408717] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [15921.038238] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15921.280559] alg: No test for adler32 (adler32-zlib) [15922.458557] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [15922.873948] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [15924.647645] Key type lgssc registered [15925.932900] Lustre: Echo OBD driver; http://www.lustre.org/ [15937.570560] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15937.597886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15939.161965] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15944.370911] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15949.280923] LustreError: 510790:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15949.296766] LustreError: 510790:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15954.302841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15954.402332] LustreError: 510791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15954.920599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15960.541778] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [15970.276448] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15976.882687] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15985.065628] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15985.632644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15993.176183] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16002.127980] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16026.382437] loop0: detected capacity change from 0 to 204800000 [16026.661990] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16026.688885] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16026.714495] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16034.980264] loop0: detected capacity change from 0 to 204800000 [16035.186258] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16035.202583] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16035.215498] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16041.878905] loop0: detected capacity change from 0 to 204800000 [16048.165121] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16048.279167] Lustre: fs15246-OST0000: new disk, initializing [16048.284829] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16048.295075] Lustre: 514411:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16048.361639] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16055.712450] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16067.257275] loop1: detected capacity change from 0 to 204800000 [16074.421077] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16074.630341] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16074.722749] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16074.804434] Lustre: fs15246-MDT0000: new disk, initializing [16075.068274] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16075.085732] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16075.167566] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16076.263131] Lustre: 515534:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d3b12d63 x1875683499366656/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788791349 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16079.965591] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16081.384244] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16081.406188] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16081.460351] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16111.291215] Lustre: server umount fs15246-MDT0000 complete [16111.598988] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16111.673377] LustreError: 510785:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16116.712170] LustreError: 510785:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16121.830929] LustreError: 510787:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16123.015531] Lustre: server umount fs15246-OST0000 complete [16138.221924] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16138.241068] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16138.261972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16142.317358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16142.318730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16142.319911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16143.879714] Lustre: server umount lustre-OST0000 complete [16149.477185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16149.495113] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16149.523590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16149.531032] Lustre: Skipped 1 previous similar message [16152.546784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16152.565088] Lustre: Skipped 1 previous similar message [16155.483079] Lustre: server umount lustre-MDT0000 complete [16157.669538] LustreError: 511733:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16157.679383] LustreError: 511733:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16159.531630] LustreError: 510770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791416 with bad export cookie 6855431418388086829 [16159.536527] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16159.555361] LustreError: 510770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16159.884303] Lustre: server umount lustre-MDT0001 complete [16174.451913] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [16177.784366] Key type lgssc unregistered [16178.096077] LNet: 518634:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16178.101803] LNetError: 518634:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16179.178306] LNet: Removed LNI 192.168.204.125@tcp [16180.465393] Key type .llcrypt unregistered [16180.467926] Key type ._llcrypt unregistered [16201.519781] Key type ._llcrypt registered [16201.526249] Key type .llcrypt registered [16201.658859] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [16202.745153] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16203.030121] alg: No test for adler32 (adler32-zlib) [16204.072269] Lustre: Lustre: Build Version: 2.17.57_103_g80273de [16204.402881] LNet: Added LNI 192.168.204.125@tcp [8/256/0/180] [16206.143238] Key type lgssc registered [16207.172451] Lustre: Echo OBD driver; http://www.lustre.org/ [16247.748864] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [16259.362537] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [16302.106901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16302.131236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16303.409824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16303.443553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16303.526619] Lustre: lustre-MDT0000: new disk, initializing [16303.631044] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16303.663270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16308.572508] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16320.176210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16320.353043] Lustre: 523115:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16320.444879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16320.450088] Lustre: Skipped 1 previous similar message [16320.597987] Lustre: lustre-MDT0001: new disk, initializing [16320.708422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16320.773433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16320.787850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16326.564987] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16336.794464] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16343.815836] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16352.458445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16352.874025] Lustre: lustre-OST0000: new disk, initializing [16352.884082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16352.897717] Lustre: 525084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16353.046424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16358.983622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16358.989453] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16359.136996] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16359.788974] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_default_debug -1 all [16370.913232] Lustre: DEBUG MARKER: oleg425-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16376.036721] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16376.212964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16380.918855] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16381.176271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16384.992612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16384.994148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16385.003844] LustreError: Skipped 1 previous similar message [16385.013250] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16385.059752] Lustre: Skipped 1 previous similar message [16389.619944] Lustre: server umount lustre-OST0000 complete [16395.246497] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16395.264210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16395.279241] Lustre: Skipped 2 previous similar messages [16400.368577] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16400.380255] Lustre: Skipped 1 previous similar message [16400.749164] Lustre: server umount lustre-MDT0000 complete [16404.625732] LustreError: 525083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791661 with bad export cookie 18430301020625460265 [16404.631761] LustreError: MGC192.168.204.125@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16404.645937] LustreError: 525083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16405.110170] Lustre: server umount lustre-MDT0001 complete [16430.688296] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing set_hostid [16441.046720] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing load_modules_local [16472.916683] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16196 sec ========= 10:35:28 (1788791728) [16474.838246] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 10:35:30 (1788791730) === [16478.610370] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 10:35:34 (1788791734) === [16513.347304] Lustre: DEBUG MARKER: oleg425-server.virtnet: executing unload_modules_local [16515.644848] Key type lgssc unregistered [16515.893475] LNet: 531725:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16515.900695] LNetError: 531725:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16516.973422] LNet: Removed LNI 192.168.204.125@tcp [16517.684269] Key type .llcrypt unregistered [16517.686593] Key type ._llcrypt unregistered