[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 467130765 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002394] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.007524] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.010101] LSM: Security Framework initializing [ 0.011049] Yama: becoming mindful. [ 0.012028] SELinux: Initializing. [ 0.013070] *** VALIDATE selinux *** [ 0.021517] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026335] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027144] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028107] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] *** VALIDATE tmpfs *** [ 0.031097] *** VALIDATE proc *** [ 0.033046] *** VALIDATE cgroup *** [ 0.034012] *** VALIDATE cgroup2 *** [ 0.036198] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037160] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040003] Spectre V2 : User space: Vulnerable [ 0.041020] Speculative Store Bypass: Vulnerable [ 0.044574] debug: unmapping init [mem 0xffffffff86a59000-0xffffffff86a60fff] [ 0.047261] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048794] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049026] ... version: 2 [ 0.050012] ... bit width: 48 [ 0.051015] ... generic registers: 4 [ 0.052016] ... value mask: 0000ffffffffffff [ 0.053018] ... max period: 00007fffffffffff [ 0.054018] ... fixed-purpose events: 3 [ 0.055013] ... event mask: 000000070000000f [ 0.057294] rcu: Hierarchical SRCU implementation. [ 0.059544] smp: Bringing up secondary CPUs ... [ 0.060657] x86: Booting SMP configuration: [ 0.061030] .... node #0, CPUs: #1 #2 #3 [ 0.066104] smp: Brought up 1 node, 4 CPUs [ 0.068019] smpboot: Max logical packages: 1 [ 0.069024] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.235527] node 0 deferred pages initialised in 164ms [ 0.239611] devtmpfs: initialized [ 0.241372] x86/mm: Memory block size: 128MB [ 0.245282] gcov: version magic: 0x41383552 [ 0.248229] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.253192] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.256434] pinctrl core: initialized pinctrl subsystem [ 0.258288] [ 0.259009] ************************************************************* [ 0.262015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.265012] ** ** [ 0.268015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.270012] ** ** [ 0.273013] ** This means that this kernel is built to expose internal ** [ 0.276015] ** IOMMU data structures, which may compromise security on ** [ 0.278013] ** your system. ** [ 0.281014] ** ** [ 0.283013] ** If you see this message and you are not debugging the ** [ 0.285011] ** kernel, report this immediately to your vendor! ** [ 0.288014] ** ** [ 0.291016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.294012] ************************************************************* [ 0.296711] NET: Registered protocol family 16 [ 0.299445] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.303055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.306103] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.312021] cpuidle: using governor menu [ 0.313628] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.314429] PCI: Using configuration type 1 for base access [ 0.316136] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.328055] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.331035] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.334117] cryptd: max_cpu_qlen set to 1000 [ 0.336060] ACPI: Added _OSI(Module Device) [ 0.337027] ACPI: Added _OSI(Processor Device) [ 0.339018] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.341014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.347813] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.355544] ACPI: Interpreter enabled [ 0.357057] ACPI: PM: (supports S0 S3 S4 S5) [ 0.359030] ACPI: Using IOAPIC for interrupt routing [ 0.361124] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.365450] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.376484] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.378042] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.381020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.386090] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.393150] acpiphp: Slot [2] registered [ 0.395129] acpiphp: Slot [5] registered [ 0.396201] acpiphp: Slot [6] registered [ 0.398103] acpiphp: Slot [7] registered [ 0.399109] acpiphp: Slot [8] registered [ 0.401169] acpiphp: Slot [9] registered [ 0.403143] acpiphp: Slot [10] registered [ 0.404136] acpiphp: Slot [3] registered [ 0.406119] acpiphp: Slot [4] registered [ 0.408100] acpiphp: Slot [11] registered [ 0.409101] acpiphp: Slot [12] registered [ 0.410103] acpiphp: Slot [13] registered [ 0.412124] acpiphp: Slot [14] registered [ 0.413100] acpiphp: Slot [15] registered [ 0.415153] acpiphp: Slot [16] registered [ 0.417093] acpiphp: Slot [17] registered [ 0.418127] acpiphp: Slot [18] registered [ 0.420243] acpiphp: Slot [19] registered [ 0.422169] acpiphp: Slot [20] registered [ 0.425112] acpiphp: Slot [21] registered [ 0.426107] acpiphp: Slot [22] registered [ 0.428136] acpiphp: Slot [23] registered [ 0.430114] acpiphp: Slot [24] registered [ 0.432110] acpiphp: Slot [25] registered [ 0.433122] acpiphp: Slot [26] registered [ 0.435104] acpiphp: Slot [27] registered [ 0.437108] acpiphp: Slot [28] registered [ 0.439083] acpiphp: Slot [29] registered [ 0.441106] acpiphp: Slot [30] registered [ 0.442104] acpiphp: Slot [31] registered [ 0.443000] PCI host bridge to bus 0000:00 [ 0.443000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.444023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.447024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.450019] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.453022] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.456026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.458161] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.462139] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.465596] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.476942] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.482052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.485019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.487017] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.491018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.494398] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.497827] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.501056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.503814] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.509017] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.522015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.528016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.535438] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.543015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.552016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.571016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.588175] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.598015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.611014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.631014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.643820] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.653014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.663015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.679014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.690842] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.698014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.705015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.723017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.733467] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.740013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.748016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.772014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.784933] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.795015] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.805016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.824015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.836017] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.839508] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.842384] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.845374] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.848232] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.854072] iommu: Default domain type: Passthrough [ 0.856326] SCSI subsystem initialized [ 0.858148] ACPI: bus type USB registered [ 0.860136] usbcore: registered new interface driver usbfs [ 0.863085] usbcore: registered new interface driver hub [ 0.865108] usbcore: registered new device driver usb [ 0.868236] pps_core: LinuxPPS API ver. 1 registered [ 0.871014] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.876084] PTP clock support registered [ 0.879122] EDAC MC: Ver: 3.0.0 [ 0.881037] PCI: Using ACPI for IRQ routing [ 0.882553] NetLabel: Initializing [ 0.884013] NetLabel: domain hash size = 128 [ 0.885009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.887121] NetLabel: unlabeled traffic allowed by default [ 0.890282] vgaarb: loaded [ 0.893272] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.896016] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.906126] clocksource: Switched to clocksource kvm-clock [ 1.017523] VFS: Disk quotas dquot_6.6.0 [ 1.019472] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.022562] *** VALIDATE ramfs *** [ 1.024246] *** VALIDATE hugetlbfs *** [ 1.026108] pnp: PnP ACPI init [ 1.028662] pnp: PnP ACPI: found 6 devices [ 1.053826] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.057651] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.060963] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.063941] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.067478] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.071180] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.074823] NET: Registered protocol family 2 [ 1.078068] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.084199] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.089470] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.095990] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.100860] TCP: Hash tables configured (established 65536 bind 65536) [ 1.105195] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.109577] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.112772] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.116309] NET: Registered protocol family 1 [ 1.119224] RPC: Registered named UNIX socket transport module. [ 1.121777] RPC: Registered udp transport module. [ 1.123573] RPC: Registered tcp transport module. [ 1.125196] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.127859] NET: Registered protocol family 44 [ 1.129836] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.132340] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.135184] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.138248] PCI: CLS 0 bytes, default 64 [ 1.141150] Unpacking initramfs... [ 2.550143] debug: unmapping init [mem 0xffff9a0e7cc54000-0xffff9a0e7ffbffff] [ 2.554794] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.557067] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.560426] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.057024] Initialise system trusted keyrings [ 3.058975] Key type blacklist registered [ 3.061107] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.070722] zbud: loaded [ 3.074097] *** VALIDATE nfs *** [ 3.075546] *** VALIDATE nfs4 *** [ 3.077464] pstore: using deflate compression [ 3.081279] Platform Keyring initialized [ 3.183498] NET: Registered protocol family 38 [ 3.186147] Key type asymmetric registered [ 3.190297] Asymmetric key parser 'x509' registered [ 3.192936] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.196666] io scheduler mq-deadline registered [ 3.198573] io scheduler kyber registered [ 3.200624] io scheduler bfq registered [ 3.202676] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.205891] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.209097] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.212673] ACPI: Power Button [PWRF] [ 3.218705] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.226390] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.241571] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.251565] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.277830] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.307805] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.335872] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.341958] Non-volatile memory driver v1.3 [ 3.343708] Linux agpgart interface v0.103 [ 3.374418] virtio_blk virtio1: [vda] 145224 512-byte logical blocks (74.4 MB/70.9 MiB) [ 3.377147] vda: detected capacity change from 0 to 74354688 [ 3.393981] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.396745] vdb: detected capacity change from 0 to 1073741824 [ 3.412428] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.415342] vdc: detected capacity change from 0 to 2621440000 [ 3.432592] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.435031] vdd: detected capacity change from 0 to 2621440000 [ 3.451348] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.454954] vde: detected capacity change from 0 to 4294967296 [ 3.478420] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.481720] vdf: detected capacity change from 0 to 4294967296 [ 3.493708] libphy: Fixed MDIO Bus: probed [ 3.499608] usbcore: registered new interface driver usbserial_generic [ 3.503058] usbserial: USB Serial support registered for generic [ 3.505511] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.509103] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.510842] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.513329] mousedev: PS/2 mouse device common for all mice [ 3.516969] rtc_cmos 00:05: RTC can wake from S4 [ 3.521120] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.521656] rtc_cmos 00:05: registered as rtc0 [ 3.529128] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.532030] intel_pstate: CPU model not supported [ 3.534625] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.537642] hid: raw HID events driver (C) Jiri Kosina [ 3.540892] usbcore: registered new interface driver usbhid [ 3.540990] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.543208] usbhid: USB HID core driver [ 3.548194] drop_monitor: Initializing network drop monitor service [ 3.551093] Initializing XFRM netlink socket [ 3.553375] NET: Registered protocol family 10 [ 3.555409] Segment Routing with IPv6 [ 3.556512] NET: Registered protocol family 17 [ 3.558269] mpls_gso: MPLS GSO support [ 3.567792] RAS: Correctable Errors collector initialized. [ 3.570428] AVX version of gcm_enc/dec engaged. [ 3.572501] AES CTR mode by8 optimization enabled [ 3.673505] sched_clock: Marking stable (3673473050, 0)->(4651238200, -977765150) [ 3.680215] registered taskstats version 1 [ 3.682853] Loading compiled-in X.509 certificates [ 3.685444] zswap: loaded using pool lzo/zbud [ 3.715197] Key type big_key registered [ 3.731304] Key type encrypted registered [ 3.733391] ima: No TPM chip found, activating TPM-bypass! [ 3.735630] ima: Allocated hash algorithm: sha1 [ 3.737758] ima: No architecture policies found [ 3.739750] evm: Initialising EVM extended attributes: [ 3.741870] evm: security.selinux [ 3.742960] evm: security.ima [ 3.744023] evm: security.capability [ 3.745569] evm: HMAC attrs: 0x1 [ 3.751835] rtc_cmos 00:05: setting system clock to 2026-06-23 18:47:06 UTC (1782240426) [ 3.759242] debug: unmapping init [mem 0xffffffff87a03000-0xffffffff87bfffff] [ 3.762434] debug: unmapping init [mem 0xffffffff86782000-0xffffffff86a58fff] [ 3.771097] Write protecting the kernel read-only data: 28672k [ 3.775055] debug: unmapping init [mem 0xffffffff84e03000-0xffffffff84ffffff] [ 3.779665] debug: unmapping init [mem 0xffffffff85714000-0xffffffff857fffff] [ 3.824945] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.833418] systemd[1]: Detected virtualization kvm. [ 3.836452] systemd[1]: Detected architecture x86-64. [ 3.838734] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.874313] systemd[1]: No hostname configured. [ 3.876574] systemd[1]: Set hostname to . [ 3.879385] random: systemd: uninitialized urandom read (16 bytes read) [ 3.882682] systemd[1]: Initializing machine ID from random generator. [ 3.934779] random: ln: uninitialized urandom read (6 bytes read) [ 4.042596] random: systemd: uninitialized urandom read (16 bytes read) [ 4.045344] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.049599] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.053990] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. [ OK ] Reached target Paths. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Setup Virtual Console... [ OK ] Reached target Timers. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.752614] device-mapper: uevent: version 1.0.3 [ 4.754946] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.577930] virtio_net virtio0 ens2: renamed from eth0 [ 5.605635] random: fast init done [ 5.708257] scsi host0: ata_piix [ 5.720785] scsi host1: ata_piix [ 5.722499] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.725263] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.426840] dracut-initqueue[582]: RTNETLINK answers: File exists [ 10.254821] random: crng init done [ 10.256340] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.820393] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.950375] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.196458] SELinux: Disabled at runtime. [ 12.259143] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.266508] systemd[1]: Detected virtualization kvm. [ 12.268026] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.765261] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.769486] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.775432] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.781242] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.785105] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.793036] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.796407] systemd[1]: Reached target rpc_pipefs.target. [ OK ] Reached target rpc_pipefs.target. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Starting udev Coldplug all Devices... [ OK ] Stopped targ[ 12.901828] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS et Initrd File Systems. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.190468] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.500668] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.536909] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.624656] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.644711] EDAC sbridge: Ver: 1.1.2 [ 15.090773] Key type dns_resolver registered [ 15.374502] NFS: Registering the id_resolver key type [ 15.376161] Key type id_resolver registered [ 15.377756] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg345-server login: [ 50.600990] libcfs: loading out-of-tree module taints kernel. [ 50.675079] Key type ._llcrypt registered [ 50.680043] Key type .llcrypt registered [ 50.861347] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 68.098913] hrtimer: interrupt took 5711801 ns [ 68.618307] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 70.109575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 70.118478] alg: No test for adler32 (adler32-zlib) [ 71.534144] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 72.322709] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 74.112971] Key type lgssc registered [ 75.634277] Lustre: Echo OBD driver; http://www.lustre.org/ [ 89.938532] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 128.797871] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 140.749269] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 140.787464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.038818] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.069192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 142.157915] Lustre: lustre-MDT0000: new disk, initializing [ 142.254663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 142.277470] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 146.431979] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.722969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 157.813196] Lustre: 6506:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 157.837755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 157.843203] Lustre: Skipped 1 previous similar message [ 157.914811] Lustre: lustre-MDT0001: new disk, initializing [ 157.979592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 157.999461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 158.040279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.379097] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 166.782832] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 174.773408] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 174.940352] Lustre: lustre-OST0000: new disk, initializing [ 174.942718] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 174.946937] Lustre: 8444:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 174.988731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 180.631679] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 182.329392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 182.343076] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 182.381319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 194.799690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 194.960696] Lustre: lustre-OST0001: new disk, initializing [ 194.964627] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 194.976695] Lustre: 9517:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 195.116559] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 202.187432] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.315540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.335582] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.400905] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 214.395494] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 221.672233] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 227.647234] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing check_logdir /tmp/testlogs/ [ 232.428536] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing yml_node [ 236.792846] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 239.354559] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 241.761267] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 243.169523] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Tue Jun 23 14:51:04 EDT 2026 [ 258.900802] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 260.258214] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 280.041536] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 280.043843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 280.057625] Lustre: Skipped 1 previous similar message [ 280.074868] Lustre: Skipped 2 previous similar messages [ 282.705079] Lustre: server umount lustre-MDT0000 complete [ 286.182857] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782240708 with bad export cookie 6415100539330952373 [ 286.193552] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 286.201646] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 290.273714] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 290.276846] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 290.277828] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 290.283625] LustreError: 6514:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 290.320173] Lustre: Skipped 3 previous similar messages [ 291.295963] LustreError: 10507:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 291.296360] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 291.311205] LustreError: 10507:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 291.331059] Lustre: Skipped 2 previous similar messages [ 292.519678] Lustre: server umount lustre-MDT0001 complete [ 295.577279] Lustre: server umount lustre-OST0000 complete [ 298.677270] Lustre: server umount lustre-OST0001 complete [ 304.265598] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 312.812349] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 345.748668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 345.994308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 346.025674] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 346.114439] Lustre: lustre-MDT0000: new disk, initializing [ 346.182409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 346.203610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 350.882571] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 360.805888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.938584] Lustre: 15067:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 361.009302] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 361.014236] Lustre: Skipped 1 previous similar message [ 361.124787] Lustre: lustre-MDT0001: new disk, initializing [ 361.246067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 361.303089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 361.323902] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 365.494675] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 374.229871] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 379.597311] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 385.708996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 386.082888] Lustre: lustre-OST0000: new disk, initializing [ 386.086356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 386.091944] Lustre: 17037:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 386.199801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 387.259963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 387.272401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 387.368477] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 391.793760] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 400.505412] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 404.720254] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 404.928575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 409.152752] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 409.367225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 413.152813] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 413.158503] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 413.165046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 413.178521] Lustre: Skipped 1 previous similar message [ 413.196759] Lustre: Skipped 1 previous similar message [ 417.526648] Lustre: server umount lustre-OST0000 complete [ 422.883222] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 422.896936] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 422.914142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 427.682644] Lustre: server umount lustre-MDT0000 complete [ 428.516833] LustreError: 15074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 428.537811] LustreError: 15074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 430.866081] LustreError: 17036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782240853 with bad export cookie 6415100539330953906 [ 430.868685] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 430.877910] LustreError: 17036:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 431.355694] Lustre: server umount lustre-MDT0001 complete [ 434.850444] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 14:54:15 (1782240855) [ 436.913655] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 438.345882] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 14:54:19 (1782240859) [ 444.020437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 444.453216] LustreError: 19446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 444.503524] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.435104] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 449.505513] LustreError: 19447:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 456.130402] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 460.793082] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 467.517554] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 472.331569] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 477.799448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 483.790344] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 491.510604] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 499.688346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 499.689924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 499.705533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 499.709900] Lustre: Skipped 2 previous similar messages [ 499.725543] Lustre: Skipped 3 previous similar messages [ 501.360503] Lustre: server umount lustre-OST0000 complete [ 510.538697] Lustre: server umount lustre-MDT0000 complete [ 513.480457] LustreError: 19427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782240936 with bad export cookie 6415100539330954788 [ 513.489626] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 513.764694] Lustre: server umount lustre-MDT0001 complete [ 524.091404] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 527.187204] Key type lgssc unregistered [ 527.466387] LNet: 22979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 527.485467] LNetError: 22979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 527.493586] LNet: Removed LNI 192.168.203.145@tcp [ 528.576139] Key type .llcrypt unregistered [ 528.583372] Key type ._llcrypt unregistered [ 549.151215] Key type ._llcrypt registered [ 549.153184] Key type .llcrypt registered [ 549.248782] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 550.473076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 550.499886] alg: No test for adler32 (adler32-zlib) [ 551.653953] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 551.877512] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 553.607186] Key type lgssc registered [ 554.682740] Lustre: Echo OBD driver; http://www.lustre.org/ [ 562.856944] loop: module loaded [ 562.970774] loop0: detected capacity change from 0 to 204800000 [ 563.214962] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 563.239873] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 563.542446] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 569.462831] loop0: detected capacity change from 0 to 204800000 [ 569.745833] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 569.773097] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 569.794757] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 574.499386] loop0: detected capacity change from 0 to 204800000 [ 580.125840] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 580.188454] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 581.587614] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 581.749358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 581.850988] Lustre: lustre-MDT0000: new disk, initializing [ 581.932326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 581.959326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 586.754316] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 595.422776] loop1: detected capacity change from 0 to 204800000 [ 601.026201] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 601.326839] Lustre: lustre-OST07c6: new disk, initializing [ 601.332978] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 601.342798] Lustre: Skipped 1 previous similar message [ 601.353269] Lustre: 26162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 601.449352] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 608.203477] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 610.957199] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 610.975984] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 611.019417] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 613.598948] Lustre: 25150:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 613.613983] Lustre: 25150:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 613.627497] Lustre: 25150:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 613.640967] Lustre: 25150:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 613.654112] Lustre: 25150:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 613.672893] Lustre: 25150:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 627.717799] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 627.730602] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 627.737015] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 627.741469] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 627.746035] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 627.751977] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 632.989465] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 632.997390] Lustre: 25151:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 633.002584] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 633.007436] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 633.013878] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 633.021776] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 633.032727] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 633.045062] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 633.063090] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 633.069377] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 633.085300] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 633.096284] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 636.366913] Lustre: 25149:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 636.387411] Lustre: 25149:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 636.394531] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 636.406527] Lustre: 25149:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 636.424651] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 636.439564] Lustre: 25149:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 636.457212] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 636.467673] Lustre: 25149:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 636.478533] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 636.487669] Lustre: 25149:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 636.501227] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 636.515355] Lustre: 25149:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 641.276905] Lustre: 26828:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 641.294525] Lustre: 26828:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 641.311296] Lustre: 26828:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 641.326821] Lustre: 26828:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 641.340566] Lustre: 26828:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 641.352163] Lustre: 26828:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 641.372615] Lustre: 26828:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 641.383164] Lustre: 26828:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 641.397445] Lustre: 26828:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 641.408959] Lustre: 26828:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 641.420089] Lustre: 26828:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 641.433236] Lustre: 26828:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 649.581682] Lustre: 26828:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 649.594288] Lustre: 26828:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 649.608361] Lustre: 26828:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 649.621155] Lustre: 26828:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 649.644215] Lustre: 26828:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 649.664267] Lustre: 26828:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 649.674275] Lustre: 26828:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 649.681653] Lustre: 26828:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 649.694030] Lustre: 26828:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 649.704765] Lustre: 26828:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 649.708946] Lustre: 26828:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 649.715511] Lustre: 26828:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 665.667921] Lustre: 26828:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 665.677314] Lustre: 26828:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 7 previous similar messages [ 665.683692] Lustre: 26828:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 665.689795] Lustre: 26828:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 665.699503] Lustre: 26828:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 665.711796] Lustre: 26828:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 665.723603] Lustre: 26828:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 665.727835] Lustre: 26828:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 665.732122] Lustre: 26828:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 665.736995] Lustre: 26828:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 665.741821] Lustre: 26828:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 665.745554] Lustre: 26828:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 702.948234] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 702.952486] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 702.973267] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 708.064355] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 713.184314] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 716.255163] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 716.401259] Lustre: server umount lustre-OST07c6 complete [ 726.274287] Lustre: server umount lustre-MDT0000 complete [ 743.808787] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 746.494419] Key type lgssc unregistered [ 746.803646] LNet: 28741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 746.811168] LNetError: 28741:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 747.877979] LNet: Removed LNI 192.168.203.145@tcp [ 748.701960] Key type .llcrypt unregistered [ 748.703956] Key type ._llcrypt unregistered [ 766.864857] Key type ._llcrypt registered [ 766.867286] Key type .llcrypt registered [ 766.951066] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 767.674952] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 767.727122] alg: No test for adler32 (adler32-zlib) [ 768.817554] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 769.079299] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 770.751225] Key type lgssc registered [ 771.703526] Lustre: Echo OBD driver; http://www.lustre.org/ [ 778.842157] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 778.865674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 780.298277] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 784.078480] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 790.500281] LustreError: 30013:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.515292] LustreError: 30013:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 792.426667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 792.737904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 797.304857] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 805.522365] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 811.590824] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 818.637450] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 819.009239] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.995729] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 826.168526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 831.977508] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 851.936588] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 851.947112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 851.978257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.964780] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 852.965623] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 852.973814] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 852.983495] LustreError: Skipped 1 previous similar message [ 853.023324] Lustre: Skipped 1 previous similar message [ 856.605264] Lustre: server umount lustre-OST0000 complete [ 863.208245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 863.211772] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 863.223889] Lustre: Skipped 1 previous similar message [ 866.690433] Lustre: server umount lustre-MDT0000 complete [ 868.323729] LustreError: 30014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 868.340115] LustreError: 30014:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 870.441541] LustreError: 29994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241293 with bad export cookie 8505320710233081087 [ 870.453068] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 870.462736] LustreError: 29994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 870.728702] Lustre: server umount lustre-MDT0001 complete [ 880.770360] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 883.517984] Key type lgssc unregistered [ 883.820928] LNet: 33760:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 883.831734] LNetError: 33760:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 883.856367] LNet: Removed LNI 192.168.203.145@tcp [ 884.730725] Key type .llcrypt unregistered [ 884.734586] Key type ._llcrypt unregistered [ 902.505970] Key type ._llcrypt registered [ 902.508667] Key type .llcrypt registered [ 902.627428] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 903.690240] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 903.842534] alg: No test for adler32 (adler32-zlib) [ 904.931456] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 905.205988] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 906.887161] Key type lgssc registered [ 908.279114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 915.380957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 915.401145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.769395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 922.189250] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 927.206663] LustreError: 35036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.215987] LustreError: 35036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 931.196225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 931.641578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 936.134996] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 943.731557] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 948.873934] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 955.118277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 955.364364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 961.097597] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 961.583350] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 969.665610] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 981.984871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 981.985653] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 981.997135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 982.010023] Lustre: Skipped 1 previous similar message [ 986.295912] Lustre: server umount lustre-OST0000 complete [ 992.226178] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.236723] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.251280] Lustre: Skipped 2 previous similar messages [ 992.269651] Lustre: Skipped 1 previous similar message [ 996.246726] Lustre: server umount lustre-MDT0000 complete [ 997.344413] LustreError: 35764:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 997.359457] LustreError: 35764:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1000.134030] LustreError: 37818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241422 with bad export cookie 1472762939111591803 [ 1000.138204] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1000.149018] LustreError: 37818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1000.540907] Lustre: server umount lustre-MDT0001 complete [ 1010.904503] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1014.223462] Key type lgssc unregistered [ 1014.506579] LNet: 38572:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1014.512331] LNetError: 38572:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1014.541834] LNet: Removed LNI 192.168.203.145@tcp [ 1015.454159] Key type .llcrypt unregistered [ 1015.457175] Key type ._llcrypt unregistered [ 1038.417934] Key type ._llcrypt registered [ 1038.422907] Key type .llcrypt registered [ 1038.526844] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1039.786996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1039.823071] alg: No test for adler32 (adler32-zlib) [ 1040.858636] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1041.201315] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1042.887137] Key type lgssc registered [ 1044.276674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1054.028887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1054.041777] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.685505] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1060.258835] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1065.956420] LustreError: 39988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1065.979316] LustreError: 39988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1068.303199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1068.802133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1072.958108] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1079.504215] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1084.765372] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1090.646857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1090.870579] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1098.154347] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1101.112631] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1106.710637] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1113.562993] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 1114.402335] Lustre: server umount lustre-MDT0000 complete [ 1115.105808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1115.110604] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.123379] LustreError: 39989:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.651867] LustreError: 39988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.655260] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.471616] LustreError: 41830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241540 with bad export cookie 716675028187359238 [ 1117.475213] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.481922] LustreError: 41830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1118.664369] LustreError: 41247:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1118.684400] LustreError: 41247:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1118.692939] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 1121.762757] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.778850] Lustre: Skipped 1 previous similar message [ 1121.783521] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1122.786220] LustreError: 39985:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1122.811523] LustreError: 39985:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1123.811829] Lustre: server umount lustre-MDT0001 complete [ 1144.381099] Lustre: server umount lustre-OST0000 complete [ 1162.263604] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1165.266846] Key type lgssc unregistered [ 1165.686143] LNet: 43847:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1165.696151] LNetError: 43847:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1165.722721] LNet: Removed LNI 192.168.203.145@tcp [ 1166.751687] Key type .llcrypt unregistered [ 1166.754071] Key type ._llcrypt unregistered [ 1189.187544] Key type ._llcrypt registered [ 1189.191204] Key type .llcrypt registered [ 1189.299188] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1190.449524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1190.556500] alg: No test for adler32 (adler32-zlib) [ 1191.677553] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1191.915351] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1193.562282] Key type lgssc registered [ 1194.673309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1202.165227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1202.207904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1203.750019] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1207.579275] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1213.928949] LustreError: 45261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1213.945389] LustreError: 45261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1214.853993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1215.173720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1219.182541] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1226.055929] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1231.013581] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1236.799600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1237.103830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1243.020357] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1248.371322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1249.812837] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1257.456349] Lustre: lustre-OST0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 1258.978752] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1258.987964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.003986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1259.018660] Lustre: Skipped 1 previous similar message [ 1262.283555] Lustre: server umount lustre-OST0000 complete [ 1282.016781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1282.025639] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.041682] Lustre: Skipped 1 previous similar message [ 1282.048833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1283.552446] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1286.180789] Lustre: server umount lustre-MDT0000 complete [ 1288.675081] LustreError: 45261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.700970] LustreError: 45261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1289.926503] LustreError: 45242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241712 with bad export cookie 17489371055623702383 [ 1289.939762] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1289.955195] LustreError: 45242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1296.349305] Lustre: server umount lustre-MDT0001 complete [ 1306.768112] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1310.364589] Key type lgssc unregistered [ 1310.739470] LNet: 48973:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1310.752791] LNetError: 48973:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1310.778324] LNet: Removed LNI 192.168.203.145@tcp [ 1311.783642] Key type .llcrypt unregistered [ 1311.788630] Key type ._llcrypt unregistered [ 1336.604339] Key type ._llcrypt registered [ 1336.615350] Key type .llcrypt registered [ 1336.752621] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1337.325497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1337.393258] alg: No test for adler32 (adler32-zlib) [ 1338.535576] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1338.753889] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1340.391187] Key type lgssc registered [ 1341.263759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1347.960218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1347.978878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1349.389996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.529774] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1359.843408] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1359.864797] LustreError: 50387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1361.729739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1362.148980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1366.491667] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1373.201742] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1378.233332] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1384.614928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1385.016744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.143298] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1395.255712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:225) [ 1399.199894] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1405.562085] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1409.072892] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1420.930792] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-2315h10m41s [ 1422.205223] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1423.842787] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1423.857072] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1423.876347] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.930344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1424.938218] Lustre: Skipped 4 previous similar messages [ 1427.418464] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 1431.009510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1431.016653] Lustre: Skipped 3 previous similar messages [ 1437.154933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1437.163509] Lustre: Skipped 3 previous similar messages [ 1438.177369] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1438.337735] Lustre: server umount lustre-MDT0000 complete [ 1439.682829] LustreError: 50382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.602796] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1443.302153] LustreError: 50384:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.325612] LustreError: 50384:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1445.838309] LustreError: 50382:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1447.835226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1448.010459] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1448.265761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1448.360113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:257) [ 1449.331811] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1452.516519] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1455.794980] Lustre: DEBUG MARKER: Wait for df (36530) ... [ 1457.322758] Lustre: DEBUG MARKER: done [ 1463.778324] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1463.786108] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1463.795032] LustreError: Skipped 1 previous similar message [ 1463.796655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1463.816340] Lustre: Skipped 3 previous similar messages [ 1463.837047] Lustre: Skipped 1 previous similar message [ 1467.853312] Lustre: server umount lustre-OST0000 complete [ 1475.044816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1475.062557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1477.895733] Lustre: server umount lustre-MDT0000 complete [ 1480.898609] LustreError: 50367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782241903 with bad export cookie 14118266422542325428 [ 1480.899414] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1480.909671] LustreError: 50367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1481.171546] Lustre: server umount lustre-MDT0001 complete [ 1490.698858] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1493.282312] Key type lgssc unregistered [ 1493.588701] LNet: 55311:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1493.597742] LNetError: 55311:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1493.617461] LNet: Removed LNI 192.168.203.145@tcp [ 1494.497679] Key type .llcrypt unregistered [ 1494.502584] Key type ._llcrypt unregistered [ 1529.274981] Key type ._llcrypt registered [ 1529.280663] Key type .llcrypt registered [ 1529.371916] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1530.597263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1530.630246] alg: No test for adler32 (adler32-zlib) [ 1531.680407] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1531.968446] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1533.655376] Key type lgssc registered [ 1534.679173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.214754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1541.254724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1542.532693] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1542.570553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1542.699355] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1546.436368] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1553.538201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1553.611211] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1553.637351] Lustre: 57514:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1553.846259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1557.442047] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1563.139198] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1568.225323] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1573.912872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1574.117615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1574.324541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1579.508822] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1584.238658] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:289) [ 1586.748229] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1593.276076] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-2315h13m34s [ 1594.542338] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1600.621490] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1602.736591] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1602.799024] LustreError: 58780:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1623.046793] Lustre: lustre-MDT0000: Client 35f4062a-02aa-465b-9d0d-a4b016062021 (at 192.168.203.45@tcp) reconnecting [ 1642.808269] LustreError: 58780:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1644.913410] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1647.134090] Lustre: DEBUG MARKER: done [ 1655.776934] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1655.790794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1655.808878] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.294197] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1656.313271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1656.321645] Lustre: Skipped 1 previous similar message [ 1658.240738] Lustre: server umount lustre-OST0000 complete [ 1666.529382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1666.530745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1666.536973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1666.559714] Lustre: Skipped 1 previous similar message [ 1667.966411] Lustre: server umount lustre-MDT0000 complete [ 1670.959754] LustreError: 57505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242093 with bad export cookie 6119504729530594369 [ 1670.971050] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.978495] LustreError: 57505:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1671.397244] Lustre: server umount lustre-MDT0001 complete [ 1681.119147] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1683.569632] Key type lgssc unregistered [ 1683.869883] LNet: 61844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1683.889057] LNetError: 61844:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1683.905204] LNet: Removed LNI 192.168.203.145@tcp [ 1684.692987] Key type .llcrypt unregistered [ 1684.696243] Key type ._llcrypt unregistered [ 1729.160234] Key type ._llcrypt registered [ 1729.163311] Key type .llcrypt registered [ 1729.264253] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 1730.181724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1730.294075] alg: No test for adler32 (adler32-zlib) [ 1731.338233] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 1731.534396] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 1733.175819] Key type lgssc registered [ 1733.967517] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1736.688725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1736.714496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.996351] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1738.020082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1738.176411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1739.106431] Lustre: Failing over lustre-MDT0000 [ 1739.376650] Lustre: server umount lustre-MDT0000 complete [ 1745.284916] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 15:16:06 (1782242166) [ 1751.891332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1752.370578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1757.063493] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1765.251566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1765.340753] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1765.368154] Lustre: 64643:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1765.627282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1769.376486] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1775.681733] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1780.799409] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1786.886444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1787.070110] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1787.243021] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1792.389653] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1793.129335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:291 to 0x280000401:321) [ 1798.639583] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1804.725809] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1811.121951] Lustre: server umount lustre-MDT0000 complete [ 1811.943420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1811.967909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1813.492685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.783472] LustreError: 64637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242237 with bad export cookie 6596826843763764268 [ 1814.789132] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1814.793276] LustreError: 64637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1815.231474] Lustre: server umount lustre-MDT0001 complete [ 1818.879891] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1824.056825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1833.280758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1834.975285] Lustre: 63568:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782242241/real 1782242241] req@0000000046ca9396 x1868816348910976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782242257 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1835.009244] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1835.016666] Lustre: Skipped 1 previous similar message [ 1836.002535] Lustre: 63569:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782242242/real 1782242242] req@000000004a81ebb2 x1868816348911104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782242258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1840.095305] Lustre: 63566:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782242246/real 1782242246] req@00000000cccca08e x1868816348911232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782242262 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1840.555225] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1840.597956] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1840.704415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.836088] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1844.784429] LustreError: 66492:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1844.794104] LustreError: 66492:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1845.488535] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1845.784112] Lustre: 63568:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782242252/real 1782242252] req@00000000269c97ef x1868816348911616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782242268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1845.815519] Lustre: 63568:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1849.823165] Lustre: 63569:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782242256/real 1782242256] req@0000000079102395 x1868816348911872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782242272 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1849.829967] LustreError: 68413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1849.858916] LustreError: 68413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1853.931574] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1853.934959] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1853.961323] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1853.985467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.280249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1858.568761] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1859.562432] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1865.836709] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1871.570749] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1880.032893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1880.041891] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.056858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1880.065652] Lustre: Skipped 1 previous similar message [ 1884.567855] Lustre: server umount lustre-MDT0000 complete [ 1885.155478] LustreError: 68910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1885.172101] LustreError: 68910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1888.333427] LustreError: 64637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242311 with bad export cookie 6596826843763766543 [ 1888.346339] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.709594] Lustre: server umount lustre-MDT0001 complete [ 1894.180487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1902.403033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.591896] Lustre: 63567:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782242313/real 1782242313] req@000000006ad268a5 x1868816348937856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782242329 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1906.638230] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.649596] Lustre: Skipped 1 previous similar message [ 1913.825367] LustreError: 63565:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000898d2b9b x1868816348938880/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1914.185936] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1914.251507] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1914.316365] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1917.772297] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1925.331166] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1930.077923] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 1930.735406] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1935.877831] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1937.484417] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1942.579738] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1950.433031] Lustre: server umount lustre-MDT0000 complete [ 1951.203186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1951.205259] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1951.213372] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1951.228052] Lustre: Skipped 2 previous similar messages [ 1951.237135] LustreError: 66491:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 1954.001338] LustreError: 64637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242376 with bad export cookie 6596826843763767922 [ 1954.020078] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1954.020447] LustreError: 64637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1954.427375] Lustre: server umount lustre-MDT0001 complete [ 1961.045630] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1964.682673] Lustre: DEBUG MARKER: files compared the same [ 1976.788878] Lustre: server umount lustre-OST0000 complete [ 1992.102579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 1995.183982] Key type lgssc unregistered [ 1995.492869] LNet: 74369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1995.503872] LNetError: 74369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1995.525450] LNet: Removed LNI 192.168.203.145@tcp [ 1996.486815] Key type .llcrypt unregistered [ 1996.488792] Key type ._llcrypt unregistered [ 2020.597345] Key type ._llcrypt registered [ 2020.600628] Key type .llcrypt registered [ 2020.696711] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2022.059270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2022.139116] alg: No test for adler32 (adler32-zlib) [ 2023.189867] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2023.438533] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2025.087141] Key type lgssc registered [ 2026.120745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2032.960429] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2032.987814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2034.432468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2039.465849] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 2044.895748] LustreError: 75785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2044.926636] LustreError: 75785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2047.371710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2047.721421] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2052.266949] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 2059.224546] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2064.360733] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2070.522552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2070.832764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.780808] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug +malloc all [ 2081.067187] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2083.544269] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2091.488336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.495666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2091.512034] Lustre: Skipped 1 previous similar message [ 2091.528026] Lustre: Skipped 1 previous similar message [ 2096.611640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2096.621530] Lustre: Skipped 1 previous similar message [ 2096.954490] Lustre: server umount lustre-OST0000 complete [ 2101.729694] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.748081] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.759568] Lustre: Skipped 1 previous similar message [ 2106.583913] Lustre: server umount lustre-MDT0000 complete [ 2106.847776] LustreError: 75780:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.866226] LustreError: 75780:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2110.151993] LustreError: 75768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242532 with bad export cookie 9334386354789400643 [ 2110.160859] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2110.162319] LustreError: 75768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2110.447963] Lustre: server umount lustre-MDT0001 complete [ 2119.553836] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2121.774431] Key type lgssc unregistered [ 2122.021228] LNet: 79316:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2122.028980] LNetError: 79316:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2123.050683] LNet: Removed LNI 192.168.203.145@tcp [ 2123.810784] Key type .llcrypt unregistered [ 2123.813389] Key type ._llcrypt unregistered [ 2144.013909] Key type ._llcrypt registered [ 2144.017288] Key type .llcrypt registered [ 2144.127232] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2145.147909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2145.211814] alg: No test for adler32 (adler32-zlib) [ 2146.253747] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2146.561861] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2148.271714] Key type lgssc registered [ 2149.594435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2157.336500] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2157.357882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2172.905196] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2178.016216] LustreError: 80686:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [ 2204.703781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2210.487196] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2217.019669] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2223.778222] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2223.945402] Lustre: *** cfs_fail_loc=706, val=0*** [ 2224.370397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2224.435214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2229.225969] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2229.737096] LustreError: 80711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2229.753354] LustreError: 80711:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2234.848598] LustreError: 80712:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.423309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.771242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2242.660477] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2249.487945] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2254.518152] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2258.405782] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2258.413323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2258.425126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2260.452127] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2260.461462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2260.463842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2262.528787] Lustre: server umount lustre-OST0000 complete [ 2265.570956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.574742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2265.581609] Lustre: Skipped 1 previous similar message [ 2265.594644] Lustre: Skipped 2 previous similar messages [ 2270.699595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2270.702569] Lustre: Skipped 1 previous similar message [ 2271.808595] Lustre: server umount lustre-MDT0000 complete [ 2274.941138] LustreError: 80693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782242697 with bad export cookie 5875912176784577079 [ 2274.948185] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2274.956265] LustreError: 80693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2275.279727] Lustre: server umount lustre-MDT0001 complete [ 2286.527226] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2289.693464] Key type lgssc unregistered [ 2289.984319] LNet: 84305:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2289.989704] LNetError: 84305:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.011638] LNet: Removed LNI 192.168.203.145@tcp [ 2290.981664] Key type .llcrypt unregistered [ 2290.987137] Key type ._llcrypt unregistered [ 2315.022852] Key type ._llcrypt registered [ 2315.025035] Key type .llcrypt registered [ 2315.115772] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2315.986311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2316.014327] alg: No test for adler32 (adler32-zlib) [ 2317.191147] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2317.410931] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2319.087179] Key type lgssc registered [ 2320.144954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2326.897552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2326.909417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.622702] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2339.372187] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2339.781839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2344.439572] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2344.928442] LustreError: 86420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2344.948699] LustreError: 86420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2350.052022] LustreError: 86423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2350.080938] LustreError: 86423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2352.854026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2353.269331] LustreError: 86420:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.285190] LustreError: 86420:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2353.300985] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2358.756726] LustreError: 87166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2358.779981] LustreError: 87166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2359.299523] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2363.880103] LustreError: 86424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2363.898546] LustreError: 86424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2367.228531] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2367.241320] Lustre: MGS: Client 096893a0-4bd3-4117-964c-12774e836b56 (at 0@lo) reconnecting [ 2367.260613] LustreError: 86434:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2367.547593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.692204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2371.806816] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2387.957456] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2387.973823] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.989497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2387.998608] Lustre: Skipped 1 previous similar message [ 2390.492983] Lustre: server umount lustre-OST0000 complete [ 2398.181847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2398.183815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2398.203846] Lustre: Skipped 1 previous similar message [ 2398.225158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2400.505357] Lustre: server umount lustre-MDT0000 complete [ 2403.302349] LustreError: 87166:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2403.331280] LustreError: 87166:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2404.350706] Lustre: server umount lustre-MDT0001 complete [ 2408.165619] Lustre: server umount MGS complete [ 2418.178451] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 15:27:19 (1782242839) [ 2442.252568] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 2452.495885] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2485.249978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2489.117848] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2497.598120] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.823861] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2497.856515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2497.909751] Lustre: lustre-MDT0001: new disk, initializing [ 2497.975310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2498.016893] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2498.028100] Lustre: 93586:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2498.040685] LustreError: 93586:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2501.434921] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2509.954839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2510.141440] Lustre: lustre-OST0000: new disk, initializing [ 2510.144888] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2510.157151] Lustre: 94351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2510.210834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2512.291841] Lustre: 94427:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000687ec297 x1868816963378304/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782242951 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2515.568341] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2523.884739] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2530.503080] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2530.511715] Lustre: MGS: Client c9673c1b-26a5-4d80-b8b3-2c648e4673b8 (at 0@lo) reconnecting [ 2530.542537] LustreError: 93572:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2530.578885] Lustre: 92855:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2530.621928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2530.716172] Lustre: lustre-MDT0000: new disk, initializing [ 2530.883671] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2530.902951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2530.935685] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2532.000193] Lustre: 95420:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000b4192fbe x1868816963385088/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1782242970 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2535.567068] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2537.121367] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2540.006283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2540.016942] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2541.104963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2541.121454] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2541.168106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2567.144163] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2567.157118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2567.171136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2567.177045] Lustre: Skipped 1 previous similar message [ 2569.706560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2569.713577] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2572.821309] Lustre: server umount lustre-OST0000 complete [ 2579.937160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2579.940876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2579.947157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2579.976662] Lustre: Skipped 1 previous similar message [ 2580.001324] Lustre: Skipped 2 previous similar messages [ 2583.037727] Lustre: server umount lustre-MDT0000 complete [ 2585.056505] LustreError: 93556:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2585.069672] LustreError: 93556:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2594.298725] Lustre: server umount lustre-MDT0001 complete [ 2598.561835] Lustre: server umount MGS complete [ 2610.474621] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 15:30:31 (1782243031) [ 2633.119250] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2636.480574] Key type lgssc unregistered [ 2636.836860] LNet: 97927:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2636.849960] LNetError: 97927:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2636.880221] LNet: Removed LNI 192.168.203.145@tcp [ 2637.917151] Key type .llcrypt unregistered [ 2637.921076] Key type ._llcrypt unregistered [ 2657.446653] Key type ._llcrypt registered [ 2657.449433] Key type .llcrypt registered [ 2657.537818] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2658.536282] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2658.554987] alg: No test for adler32 (adler32-zlib) [ 2659.595958] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2659.869418] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2661.519150] Key type lgssc registered [ 2662.680241] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2669.987086] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2670.016098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2671.435930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2676.482123] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2681.824565] LustreError: 99151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.842955] LustreError: 99151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2684.441387] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2685.043832] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2689.494169] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2697.052551] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2702.066463] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2708.805898] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2709.170105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2715.893885] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2716.342817] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 2724.510513] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2732.001729] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2732.002972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2732.004897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2732.008925] Lustre: Skipped 1 previous similar message [ 2737.124431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2737.130868] Lustre: Skipped 2 previous similar messages [ 2742.245238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.257819] Lustre: Skipped 1 previous similar message [ 2746.335909] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2746.510602] Lustre: server umount lustre-OST0000 complete [ 2751.456649] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2751.467519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2751.488925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2752.480958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.601196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2757.611360] Lustre: Skipped 3 previous similar messages [ 2764.255119] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2764.410960] Lustre: server umount lustre-MDT0000 complete [ 2767.781194] LustreError: 99132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243190 with bad export cookie 18418446438738584137 [ 2767.788223] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2767.792657] LustreError: 99132:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2768.106115] Lustre: server umount lustre-MDT0001 complete [ 2778.180207] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2780.597373] Key type lgssc unregistered [ 2780.882611] LNet: 102684:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2780.887872] LNetError: 102684:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2780.898310] LNet: Removed LNI 192.168.203.145@tcp [ 2781.709649] Key type .llcrypt unregistered [ 2781.712153] Key type ._llcrypt unregistered [ 2797.854905] Key type ._llcrypt registered [ 2797.860934] Key type .llcrypt registered [ 2797.972454] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2798.887709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2799.115751] alg: No test for adler32 (adler32-zlib) [ 2800.241078] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2800.455917] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2802.167158] Key type lgssc registered [ 2803.421504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2808.991984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2809.007378] LustreError: 103764:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2809.038931] LustreError: 103768:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2809.046673] LustreError: 103764:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4978 [ 2809.056469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2809.060431] LustreError: 103764:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2809.060455] LustreError: 103764:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2810.513206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.832600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.180218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2822.658405] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2828.180799] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing lsmod [ 2830.548115] LustreError: 104974:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2830.559743] LustreError: 104975:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2830.568233] LustreError: 104974:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 2830.577111] LustreError: 104974:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2830.580643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2830.586828] LustreError: 104974:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2830.958287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2833.380891] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2833.387330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2833.403621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.457286] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2836.457440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2836.476109] Lustre: Skipped 1 previous similar message [ 2836.505913] Lustre: Skipped 2 previous similar messages [ 2839.434608] Lustre: server umount lustre-MDT0000 complete [ 2841.576325] LustreError: 103802:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2841.599188] LustreError: 103802:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2842.595079] LustreError: 103805:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2842.612827] LustreError: 103805:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2842.686748] LustreError: 103783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243265 with bad export cookie 14014470845616534232 [ 2842.691952] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2842.706201] LustreError: 104237:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000fcce778f x1868817469819392/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2842.711695] LustreError: 103783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2842.724859] Lustre: 103797:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.203.145@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2843.151064] Lustre: server umount lustre-MDT0001 complete [ 2857.125882] Lustre: server umount lustre-OST0000 complete [ 2864.017342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2864.352275] LustreError: 106037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2864.410231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2867.664140] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2869.728992] LustreError: 106038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2874.857047] LustreError: 106037:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2875.237438] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.564523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2879.742672] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2887.045514] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2892.996146] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2899.689874] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2899.995852] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2906.855917] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 2910.283288] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 2914.142604] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2920.420178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2920.430172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2920.435092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2920.441105] Lustre: Skipped 1 previous similar message [ 2920.452078] Lustre: Skipped 1 previous similar message [ 2925.541228] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2925.546869] Lustre: Skipped 1 previous similar message [ 2926.496404] Lustre: server umount lustre-OST0000 complete [ 2930.658972] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2930.698071] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2930.707150] Lustre: Skipped 1 previous similar message [ 2935.776186] LustreError: 106038:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2935.797731] LustreError: 106038:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2936.093223] Lustre: server umount lustre-MDT0000 complete [ 2939.560626] LustreError: 106019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243362 with bad export cookie 14014470845616534911 [ 2939.568924] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2939.580113] LustreError: 106019:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2939.807894] Lustre: server umount lustre-MDT0001 complete [ 2950.137435] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 2952.885318] Key type lgssc unregistered [ 2953.155309] LNet: 109571:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2953.160634] LNetError: 109571:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2954.219714] LNet: Removed LNI 192.168.203.145@tcp [ 2955.365082] Key type .llcrypt unregistered [ 2955.374763] Key type ._llcrypt unregistered [ 2977.772935] Key type ._llcrypt registered [ 2977.775308] Key type .llcrypt registered [ 2977.874993] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 2978.928994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2978.962734] alg: No test for adler32 (adler32-zlib) [ 2980.028614] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 2980.284835] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 2981.999326] Key type lgssc registered [ 2983.110504] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2990.675219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2990.702717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2992.072219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2996.589992] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3001.316283] LustreError: 110988:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3001.339181] LustreError: 110988:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3004.196191] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.636420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3009.194229] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3016.995982] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3022.663532] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3028.886804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3029.319943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3034.840327] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3036.543384] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3041.579970] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3047.366563] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3050.617702] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3056.108561] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3056.121183] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3056.139685] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3056.432170] Lustre: server umount lustre-OST0000 complete [ 3061.217045] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3061.236834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3061.257191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.244122] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3062.270458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3062.275908] Lustre: Skipped 1 previous similar message [ 3067.362237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3067.366221] Lustre: Skipped 1 previous similar message [ 3070.509451] Lustre: server umount lustre-MDT0000 complete [ 3071.465577] LustreError: 110987:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3071.490178] LustreError: 110987:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3074.088351] LustreError: 110968:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243496 with bad export cookie 13079435808381607366 [ 3074.099430] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3074.544838] Lustre: server umount lustre-MDT0001 complete [ 3087.073070] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3090.198610] Key type lgssc unregistered [ 3090.599327] LNet: 114623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3090.606944] LNetError: 114623:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3091.629895] LNet: Removed LNI 192.168.203.145@tcp [ 3092.772364] Key type .llcrypt unregistered [ 3092.775296] Key type ._llcrypt unregistered [ 3111.577647] Key type ._llcrypt registered [ 3111.580426] Key type .llcrypt registered [ 3111.653527] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3112.816569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3112.841118] alg: No test for adler32 (adler32-zlib) [ 3113.955788] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 3114.197087] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3115.903187] Key type lgssc registered [ 3117.127730] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3125.493872] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3125.511256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3127.002320] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3131.563074] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3136.499185] LustreError: 115845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3136.522055] LustreError: 115845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3139.989691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3140.373423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3144.955242] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3153.397900] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3159.174333] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3166.146268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3166.603025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.762667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3173.386711] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3180.418063] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3192.294018] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3192.314816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3192.324614] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3196.388842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3196.399318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3196.421339] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3197.625992] Lustre: server umount lustre-OST0000 complete [ 3201.505355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3201.515352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3201.531510] Lustre: Skipped 1 previous similar message [ 3201.552902] Lustre: Skipped 2 previous similar messages [ 3206.626224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3206.633038] Lustre: Skipped 1 previous similar message [ 3207.767845] Lustre: server umount lustre-MDT0000 complete [ 3211.332954] LustreError: 115825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243634 with bad export cookie 15652144181364988258 [ 3211.335069] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3211.342938] LustreError: 115825:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3211.735952] Lustre: server umount lustre-MDT0001 complete [ 3222.473410] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3225.240985] Key type lgssc unregistered [ 3225.531868] LNet: 119376:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3225.541780] LNetError: 119376:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3225.558764] LNet: Removed LNI 192.168.203.145@tcp [ 3226.450249] Key type .llcrypt unregistered [ 3226.452698] Key type ._llcrypt unregistered [ 3248.446489] Key type ._llcrypt registered [ 3248.450185] Key type .llcrypt registered [ 3248.560449] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3249.699421] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3249.743928] alg: No test for adler32 (adler32-zlib) [ 3250.813605] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 3251.125229] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3252.848243] Key type lgssc registered [ 3253.920284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3261.168173] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3261.183851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3262.646591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.762640] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3273.189669] LustreError: 120793:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3273.213848] LustreError: 120793:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3275.456086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3275.938746] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3280.122962] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3287.718951] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3293.874590] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3300.987233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3301.467284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.651147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3308.684950] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3316.749813] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3327.512561] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3332.145317] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3332.154723] Lustre: Skipped 2 previous similar messages [ 3335.819294] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3335.869613] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3337.332528] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3337.345307] Lustre: Skipped 2 previous similar messages [ 3341.876443] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3341.880581] Lustre: Skipped 1 previous similar message [ 3348.896589] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.45@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3348.904970] Lustre: Skipped 2 previous similar messages [ 3356.734603] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.45@tcp 192.168.0.[2,10]@tcp [ 3362.463311] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.45/24@tcp in log lustre-MDT0000 [ 3362.473446] Lustre: Skipped 2 previous similar messages [ 3367.063350] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.45/24@tcp [ 3367.065993] Lustre: Skipped 1 previous similar message [ 3374.561834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3374.577540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3374.598686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.660518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3377.662161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3377.662185] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.711094] Lustre: Skipped 1 previous similar message [ 3378.972785] Lustre: server umount lustre-OST0000 complete [ 3387.873866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3387.883706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.887191] Lustre: Skipped 1 previous similar message [ 3387.913618] Lustre: Skipped 1 previous similar message [ 3389.897913] Lustre: server umount lustre-MDT0000 complete [ 3392.993579] LustreError: 120789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3393.028627] LustreError: 120789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3394.375439] LustreError: 120775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243817 with bad export cookie 12923265350024562258 [ 3394.379319] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3394.387460] LustreError: 120775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3394.684911] Lustre: server umount lustre-MDT0001 complete [ 3406.642438] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3409.546974] Key type lgssc unregistered [ 3409.844818] LNet: 125743:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3409.854521] LNetError: 125743:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3409.867504] LNet: Removed LNI 192.168.203.145@tcp [ 3411.052203] Key type .llcrypt unregistered [ 3411.054901] Key type ._llcrypt unregistered [ 3441.199239] Key type ._llcrypt registered [ 3441.201943] Key type .llcrypt registered [ 3441.330985] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3442.486328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3442.566069] alg: No test for adler32 (adler32-zlib) [ 3443.712526] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 3443.950948] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3445.631988] Key type lgssc registered [ 3446.819498] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3454.132057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3454.148866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3455.597432] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3455.614395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3455.621023] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.45/24@tcp [ 3459.420147] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3466.215470] LustreError: 127305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3466.234590] LustreError: 127305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3467.769594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3468.091337] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3468.114158] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.45/24@tcp [ 3468.199475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3472.449638] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3478.273746] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3483.377829] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3490.809199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3491.040620] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3497.723530] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3500.301028] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3505.706899] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3515.876459] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3515.877462] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3515.892914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3515.899915] Lustre: Skipped 1 previous similar message [ 3515.905917] Lustre: Skipped 1 previous similar message [ 3520.993751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3521.009651] Lustre: Skipped 1 previous similar message [ 3521.359802] Lustre: server umount lustre-OST0000 complete [ 3526.114562] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3526.119038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3526.134643] Lustre: Skipped 1 previous similar message [ 3526.146900] Lustre: Skipped 1 previous similar message [ 3531.234589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3531.254495] Lustre: Skipped 1 previous similar message [ 3533.398846] Lustre: server umount lustre-MDT0000 complete [ 3536.354566] LustreError: 127304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3536.381747] LustreError: 127304:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3537.544741] LustreError: 127285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782243960 with bad export cookie 12464430441337468357 [ 3537.553531] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3537.556826] LustreError: 127285:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3537.869587] Lustre: server umount lustre-MDT0001 complete [ 3549.029626] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3552.193478] Key type lgssc unregistered [ 3552.530704] LNet: 130891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3552.546669] LNetError: 130891:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3552.569345] LNet: Removed LNI 192.168.203.145@tcp [ 3553.380951] Key type .llcrypt unregistered [ 3553.383950] Key type ._llcrypt unregistered [ 3577.201816] Key type ._llcrypt registered [ 3577.204578] Key type .llcrypt registered [ 3577.331460] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3591.329675] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3592.397617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3592.431976] alg: No test for adler32 (adler32-zlib) [ 3593.444338] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 3593.705890] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3595.375160] Key type lgssc registered [ 3596.385142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3625.827666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3625.879364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3627.068429] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3627.094828] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3627.169957] Lustre: lustre-MDT0000: new disk, initializing [ 3627.263157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3627.281333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3631.469874] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3641.256534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.405408] Lustre: 134291:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3641.475346] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3641.483218] Lustre: Skipped 1 previous similar message [ 3641.557364] Lustre: lustre-MDT0001: new disk, initializing [ 3641.668670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3641.707725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3641.715106] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3645.816670] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3654.196396] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3659.064902] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3665.015296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3665.277205] Lustre: lustre-OST0000: new disk, initializing [ 3665.280231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3665.285626] Lustre: 136261:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3665.370864] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3671.342590] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3674.156720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3674.173437] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3674.290275] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3679.848550] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3684.652483] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3684.893716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3689.541105] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3689.805414] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3695.076902] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3695.083933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3695.088204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3695.098398] Lustre: Skipped 1 previous similar message [ 3697.809318] Lustre: server umount lustre-OST0000 complete [ 3703.271621] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3703.281196] LustreError: Skipped 1 previous similar message [ 3703.286988] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3703.301312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3703.306833] Lustre: Skipped 1 previous similar message [ 3705.314980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.316344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3705.348272] Lustre: Skipped 1 previous similar message [ 3707.770528] Lustre: server umount lustre-MDT0000 complete [ 3711.565580] LustreError: 136260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244134 with bad export cookie 11905084809885437171 [ 3711.575693] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3712.004272] Lustre: server umount lustre-MDT0001 complete [ 3719.589681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3720.091280] LustreError: 138477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3720.114858] LustreError: 138477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3720.176648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3725.007440] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3725.290648] LustreError: 138478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3729.385430] LustreError: 138477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3734.500587] LustreError: 138478:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3734.963625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3735.432329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3740.705872] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3748.731842] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3754.378054] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3760.859682] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3761.057224] LustreError: 140299:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3761.279481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3767.455663] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3774.521971] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3779.285706] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3779.505750] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3784.346594] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3784.594453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3794.292709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3794.449940] Lustre: lustre-OST0001: new disk, initializing [ 3794.453311] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3794.459797] Lustre: 141851:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3794.547394] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3800.749248] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 3802.269242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3802.293851] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3802.404621] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3811.292727] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3811.529299] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3817.868734] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3818.221476] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3824.875512] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff969383821000.ost_server_uuid 50 [ 3826.451918] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff969383821000.ost_server_uuid in IDLE state after 0 sec [ 3838.441895] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3838.444217] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.450093] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3838.462846] Lustre: Skipped 1 previous similar message [ 3840.489038] Lustre: server umount lustre-OST0001 complete [ 3843.556646] LustreError: 140300:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3843.600765] LustreError: 140300:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3846.635867] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3846.642345] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3846.645746] LustreError: Skipped 2 previous similar messages [ 3846.665970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3846.679346] Lustre: Skipped 2 previous similar messages [ 3858.399695] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3858.687785] Lustre: server umount lustre-OST0000 complete [ 3863.524822] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3863.539644] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3863.557000] Lustre: Skipped 1 previous similar message [ 3863.562978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3863.567870] Lustre: Skipped 4 previous similar messages [ 3876.319318] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3876.556427] Lustre: server umount lustre-MDT0000 complete [ 3879.398401] LustreError: 138477:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3879.427498] LustreError: 138477:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3880.071048] LustreError: 138459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244302 with bad export cookie 11905084809885438053 [ 3880.082326] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.450091] Lustre: server umount lustre-MDT0001 complete [ 3900.332939] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 3903.003453] Key type lgssc unregistered [ 3903.363985] LNet: 144854:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3903.374718] LNetError: 144854:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3903.399200] LNet: Removed LNI 192.168.203.145@tcp [ 3904.197626] Key type .llcrypt unregistered [ 3904.201487] Key type ._llcrypt unregistered [ 3944.038470] Key type ._llcrypt registered [ 3944.040991] Key type .llcrypt registered [ 3944.117945] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 3959.166243] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 3960.346741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3960.550742] alg: No test for adler32 (adler32-zlib) [ 3961.584870] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 3961.831885] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 3963.535223] Key type lgssc registered [ 3964.692852] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3998.624658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3998.691159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3999.966728] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3999.996456] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4000.093629] Lustre: lustre-MDT0000: new disk, initializing [ 4000.151424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4000.163936] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4004.789048] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4015.128106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4015.268781] Lustre: 149010:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4015.348901] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4015.354929] Lustre: Skipped 1 previous similar message [ 4015.505877] Lustre: lustre-MDT0001: new disk, initializing [ 4015.600433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4015.627043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4015.648818] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4019.891981] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4029.147497] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4034.491427] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4040.923609] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4041.178237] Lustre: lustre-OST0000: new disk, initializing [ 4041.184310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4041.190739] Lustre: 150978:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4041.251844] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4047.627710] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4049.973226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4049.989284] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4050.132042] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4057.486755] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4062.877120] Lustre: Failing over lustre-OST0000 [ 4063.136884] Lustre: server umount lustre-OST0000 complete [ 4080.607101] Lustre: 147234:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782244487/real 1782244487] req@0000000092ad7a2c x1868818687230976/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1782244503 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4080.627297] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.592320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4081.695143] Lustre: 147235:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782244488/real 1782244488] req@000000009bddfab2 x1868818687231488/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782244504 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4081.698977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4081.760640] Lustre: 147235:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4082.017606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4082.033886] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4085.033737] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4086.753392] Lustre: 147234:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782244493/real 1782244493] req@000000004122d6b9 x1868818687233152/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782244509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4086.795983] Lustre: 147234:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4088.970377] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4091.898705] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4091.900228] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4092.815024] Lustre: Failing over lustre-MDT0000 [ 4093.045070] Lustre: server umount lustre-MDT0000 complete [ 4093.856140] Lustre: 147235:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782244498/real 1782244498] req@00000000a8727db2 x1868818687234176/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782244514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4093.881797] Lustre: 147235:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4096.484031] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.487414] LustreError: 149017:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4096.502903] Lustre: Skipped 1 previous similar message [ 4096.534860] LustreError: 149017:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4096.988536] LustreError: 150172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.605290] LustreError: 149015:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4101.627436] LustreError: 149015:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4106.729737] LustreError: 149020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4106.755813] LustreError: 149020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4111.862983] LustreError: 150171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4111.899539] LustreError: 150171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4112.253985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4112.376775] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4112.683226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.725593] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4113.417524] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4117.031804] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4118.008141] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4118.012769] Lustre: Skipped 1 previous similar message [ 4118.030508] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4118.073976] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4127.719491] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4127.725745] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4127.735748] Lustre: Skipped 1 previous similar message [ 4127.745595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4130.457832] Lustre: server umount lustre-OST0000 complete [ 4138.464410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4138.467753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4138.478157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4138.485790] Lustre: Skipped 2 previous similar messages [ 4138.490120] Lustre: Skipped 2 previous similar messages [ 4139.919805] Lustre: server umount lustre-MDT0000 complete [ 4143.482753] LustreError: 149803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782244566 with bad export cookie 9730790626999177479 [ 4143.485412] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4143.498861] LustreError: 149803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4143.987203] Lustre: server umount lustre-MDT0001 complete [ 4155.722485] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 4158.552909] Key type lgssc unregistered [ 4158.866606] LNet: 154630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4158.879119] LNetError: 154630:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4158.914059] LNet: Removed LNI 192.168.203.145@tcp [ 4159.866117] Key type .llcrypt unregistered [ 4159.868989] Key type ._llcrypt unregistered [ 4195.857961] Key type ._llcrypt registered [ 4195.859764] Key type .llcrypt registered [ 4195.974274] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4197.300427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4197.393667] alg: No test for adler32 (adler32-zlib) [ 4198.517775] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 4198.775193] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 4200.475086] Key type lgssc registered [ 4201.464595] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4208.633845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4208.662329] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.237590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4215.577989] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4219.364986] LustreError: 156151:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4219.393456] LustreError: 156151:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4224.504484] LustreError: 156152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4225.064324] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4228.727762] LustreError: 156180:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4228.744741] LustreError: 156180:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4228.752804] LustreError: 156180:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 4228.996138] Lustre: server umount lustre-MDT0000 complete [ 4244.578742] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4255.216355] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4290.844213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4291.088271] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4291.114797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4291.205565] Lustre: lustre-MDT0000: new disk, initializing [ 4291.317900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4291.348577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4296.213369] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4307.224076] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4307.406779] Lustre: 159483:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4307.535351] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4307.545241] Lustre: Skipped 1 previous similar message [ 4307.696772] Lustre: lustre-MDT0001: new disk, initializing [ 4307.835894] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4307.876864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4307.889557] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4313.180370] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4322.240247] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4327.884868] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4335.164961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4335.349618] Lustre: lustre-OST0000: new disk, initializing [ 4335.352826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4335.357777] Lustre: 161453:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4335.432373] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4341.389149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4341.402113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4341.556882] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4341.853759] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4352.407598] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4357.595603] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4359.584687] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 15:59:40 (1782244780) [ 4589.538792] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4589.553064] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.568811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4592.609613] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4592.630106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4592.630106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4592.636788] Lustre: Skipped 1 previous similar message [ 4592.668185] Lustre: Skipped 1 previous similar message [ 4595.700050] Lustre: server umount lustre-MDT0000 complete [ 4597.737098] LustreError: 159490:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4597.759466] LustreError: 159490:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4599.271972] LustreError: 159478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245022 with bad export cookie 7116854241529002681 [ 4599.273689] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4599.288124] LustreError: 159478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4602.847844] LustreError: 163235:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4602.848546] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4602.897065] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4605.703348] Lustre: server umount lustre-MDT0001 complete [ 4624.353485] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4624.480379] Lustre: server umount lustre-OST0000 complete [ 4635.395375] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4646.072323] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4680.574231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4680.749661] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4680.780179] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4680.870789] Lustre: lustre-MDT0000: new disk, initializing [ 4681.003629] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.042802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4685.076840] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4694.594964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4694.704279] Lustre: 166123:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4694.790942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4694.804438] Lustre: Skipped 1 previous similar message [ 4694.934528] Lustre: lustre-MDT0001: new disk, initializing [ 4695.024488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4695.064320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4695.083828] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4699.237915] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4708.164039] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4713.415103] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4720.035722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4720.354964] Lustre: lustre-OST0000: new disk, initializing [ 4720.358283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4720.363368] Lustre: 168090:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4720.480765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4726.868326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4726.889567] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4726.896124] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4726.993502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4736.142445] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4740.245352] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4740.449578] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4743.911880] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4744.075412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4747.744437] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4747.757642] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4747.759195] LustreError: Skipped 1 previous similar message [ 4747.761816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4747.761824] Lustre: Skipped 1 previous similar message [ 4747.800399] Lustre: Skipped 1 previous similar message [ 4751.740628] Lustre: server umount lustre-OST0000 complete [ 4756.459743] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4756.473065] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.488326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4756.497893] Lustre: Skipped 1 previous similar message [ 4761.948276] Lustre: server umount lustre-MDT0000 complete [ 4763.115160] LustreError: 169213:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4763.143843] LustreError: 169213:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4765.941569] LustreError: 166115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245188 with bad export cookie 7116854241529161511 [ 4765.946178] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4765.957730] LustreError: 166115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4766.383597] Lustre: server umount lustre-MDT0001 complete [ 4774.530955] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:06:35 (1782245195) [ 4792.086528] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4802.177883] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 4841.498732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4841.791771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4841.819993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4841.892254] Lustre: lustre-MDT0000: new disk, initializing [ 4841.968590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4841.994231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4846.410895] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4858.488490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.797560] Lustre: 172847:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4859.146791] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4864.928976] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4874.320605] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4879.754675] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4886.616081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4886.876119] Lustre: lustre-OST0000: new disk, initializing [ 4886.880270] Lustre: Skipped 1 previous similar message [ 4886.884615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4886.891307] Lustre: Skipped 2 previous similar messages [ 4886.897413] Lustre: 174824:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4892.879381] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 4895.299522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4895.318599] Lustre: Skipped 1 previous similar message [ 4895.327846] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4895.427784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4901.989840] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4911.072175] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4911.078481] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.083302] LustreError: Skipped 1 previous similar message [ 4911.086421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4911.104308] Lustre: Skipped 2 previous similar messages [ 4911.120214] Lustre: Skipped 3 previous similar messages [ 4923.359442] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4923.600493] Lustre: server umount lustre-OST0000 complete [ 4931.040391] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4931.054481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4933.466667] Lustre: server umount lustre-MDT0000 complete [ 4936.680218] LustreError: 172859:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4936.702273] LustreError: 172859:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4936.892644] LustreError: 172841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245359 with bad export cookie 7116854241529162393 [ 4936.900334] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4936.903732] LustreError: 172841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4937.303729] Lustre: server umount lustre-MDT0001 complete [ 4944.957802] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 16:09:25 (1782245365) [ 4960.590867] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 4970.730836] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5005.272471] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.459919] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5005.500474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5005.584882] Lustre: lustre-MDT0000: new disk, initializing [ 5005.696368] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5005.705940] Lustre: Skipped 2 previous similar messages [ 5005.735533] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5010.338967] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5020.636582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5020.750514] Lustre: 179146:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5021.023610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5025.136346] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5034.481684] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5040.377495] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5046.625909] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5046.940418] Lustre: 181124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5051.752710] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5056.089410] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5056.221667] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5059.364630] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5066.731895] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5066.748285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5066.771125] Lustre: Skipped 1 previous similar message [ 5066.781406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5066.786772] Lustre: Skipped 7 previous similar messages [ 5071.288290] Lustre: server umount lustre-OST0000 complete [ 5080.449196] LustreError: 182228:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x70616d65646f6e:0x4:0x0].0x0 (00000000d30c00a1) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 5082.080426] LustreError: 179952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5082.100879] LustreError: 179952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5082.511425] Lustre: server umount lustre-MDT0000 complete [ 5085.296133] LustreError: 179138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245508 with bad export cookie 7116854241529163478 [ 5085.297814] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5085.302163] LustreError: 179138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5085.534380] Lustre: server umount lustre-MDT0001 complete [ 5094.273272] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5096.829407] Key type lgssc unregistered [ 5097.066797] LNet: 182979:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5097.079352] LNetError: 182979:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5097.114957] LNet: Removed LNI 192.168.203.145@tcp [ 5097.822573] Key type .llcrypt unregistered [ 5097.829515] Key type ._llcrypt unregistered [ 5120.506773] Key type ._llcrypt registered [ 5120.509666] Key type .llcrypt registered [ 5120.624925] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5121.646033] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5121.767337] alg: No test for adler32 (adler32-zlib) [ 5122.829767] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 5123.037701] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5124.697043] Key type lgssc registered [ 5125.825435] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5133.635545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5133.679695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5135.226898] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5139.611616] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5144.552989] LustreError: 184395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5144.586277] LustreError: 184395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5147.596598] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5147.986057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5151.941363] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5158.032618] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5163.528687] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5169.910885] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5170.207132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5175.932589] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5179.444610] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5183.195588] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5202.915507] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5202.933419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5202.958212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5204.967465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5204.971382] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5204.971533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.979595] Lustre: Skipped 1 previous similar message [ 5206.938717] Lustre: server umount lustre-OST0000 complete [ 5214.689634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5214.698484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.715050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5216.879333] Lustre: server umount lustre-MDT0000 complete [ 5220.322252] LustreError: 184395:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.352754] LustreError: 184395:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5220.466207] LustreError: 184375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245643 with bad export cookie 5049119307411500090 [ 5220.475476] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5220.484975] LustreError: 184375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5220.883712] Lustre: server umount lustre-MDT0001 complete [ 5231.159112] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 5233.688455] Key type lgssc unregistered [ 5233.984544] LNet: 187948:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5233.997112] LNetError: 187948:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5234.010804] LNet: Removed LNI 192.168.203.145@tcp [ 5235.039828] Key type .llcrypt unregistered [ 5235.042370] Key type ._llcrypt unregistered [ 5260.773716] Key type ._llcrypt registered [ 5260.780854] Key type .llcrypt registered [ 5261.013946] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5261.844386] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5261.853223] alg: No test for adler32 (adler32-zlib) [ 5263.065714] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 5263.419168] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 5265.127171] Key type lgssc registered [ 5266.057260] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5276.017657] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5276.077375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5278.069834] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5283.857956] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5287.398887] LustreError: 189367:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5287.415080] LustreError: 189367:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5292.525821] LustreError: 189366:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5293.182233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5293.606472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5297.921775] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5305.400446] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5311.168928] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5318.000718] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5318.425558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.558513] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5324.644616] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5332.115651] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5340.129720] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5340.140198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5340.162805] Lustre: lustre-OST0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 5340.184779] Lustre: Skipped 1 previous similar message [ 5343.206200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5343.207975] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5343.212847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5345.044735] Lustre: server umount lustre-OST0000 complete [ 5352.680806] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff96938512d800.ost_server_uuid 50 [ 5354.284856] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5368.483114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5368.496991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5368.499914] Lustre: Skipped 1 previous similar message [ 5373.837857] Lustre: server umount lustre-MDT0000 complete [ 5373.922385] LustreError: 190386:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5373.947163] LustreError: 190386:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5377.165352] LustreError: 189346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245799 with bad export cookie 10406428575251246758 [ 5377.169807] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5377.177686] LustreError: 189346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5377.492948] Lustre: server umount lustre-MDT0001 complete [ 5385.630901] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 16:16:46 (1782245806) [ 5392.097312] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5392.812974] LustreError: 193283:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5392.922857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5397.799826] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5397.985955] LustreError: 193284:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5406.331783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.684544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5411.146980] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5417.640912] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5422.690156] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5429.124418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5429.442639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5435.317113] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5437.623197] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5442.204714] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5448.347477] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5448.539742] Lustre: lustre-OST0001: new disk, initializing [ 5448.543583] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5448.547922] Lustre: 196124:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5448.626286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5454.271753] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5457.496974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5457.503497] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5457.620927] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5462.468785] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5467.629485] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5467.635040] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5467.654370] Lustre: Skipped 1 previous similar message [ 5467.666520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5473.219359] Lustre: lustre-OST0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 5473.233484] Lustre: Skipped 3 previous similar messages [ 5479.391839] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5479.504954] Lustre: server umount lustre-OST0000 complete [ 5483.460798] LustreError: 195106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5483.481133] LustreError: 195106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5484.751192] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5484.916109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5488.094399] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5488.299202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5503.970765] LustreError: 195106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.971529] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5503.986584] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5503.991585] LustreError: 195106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 12 previous similar messages [ 5503.999708] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5504.000872] Lustre: Skipped 2 previous similar messages [ 5504.034902] Lustre: Skipped 6 previous similar messages [ 5508.839537] Lustre: server umount lustre-OST0001 complete [ 5514.209066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5514.212213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5514.219737] LustreError: Skipped 1 previous similar message [ 5514.257163] Lustre: Skipped 1 previous similar message [ 5523.428406] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5523.434313] Lustre: Skipped 5 previous similar messages [ 5527.008532] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5527.217119] Lustre: server umount lustre-MDT0000 complete [ 5530.191294] LustreError: 193264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782245952 with bad export cookie 10406428575251247829 [ 5530.202055] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5530.203255] LustreError: 193264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5530.423069] Lustre: server umount lustre-MDT0001 complete [ 5549.287820] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 16:19:30 (1782245970) [ 5554.724905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5555.005369] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5555.039696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5555.197139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5557.960649] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5564.402557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5564.541685] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5564.598977] Lustre: 199563:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5569.812898] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5576.273467] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5581.972842] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5587.821077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5587.979387] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5593.633991] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5596.779484] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5600.657826] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5607.210960] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5607.350489] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5613.412894] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5619.094777] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5622.240584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5622.245214] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5622.265879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5627.517699] Lustre: server umount lustre-OST0000 complete [ 5629.900895] LustreError: 201413:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5629.926226] LustreError: 201413:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5645.797197] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5645.801545] LustreError: Skipped 2 previous similar messages [ 5647.812554] Lustre: server umount lustre-OST0001 complete [ 5656.049620] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5656.059702] Lustre: Skipped 4 previous similar messages [ 5657.271596] Lustre: server umount lustre-MDT0000 complete [ 5660.594489] LustreError: 199556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246083 with bad export cookie 10406428575251249236 [ 5660.599871] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5660.607976] LustreError: 199556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5660.874425] Lustre: server umount lustre-MDT0001 complete [ 5680.685868] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 16:21:41 (1782246101) [ 5694.400853] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 5702.627931] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 5733.190985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5733.523455] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5733.609599] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5733.662077] Lustre: lustre-MDT0000: new disk, initializing [ 5733.770876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5733.775930] Lustre: Skipped 3 previous similar messages [ 5733.789530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5738.767435] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5750.006239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5750.089408] Lustre: 207658:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5750.153495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5750.156508] Lustre: Skipped 1 previous similar message [ 5750.228892] Lustre: lustre-MDT0001: new disk, initializing [ 5750.376898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5750.391311] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5754.235517] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5762.490409] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5768.152222] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5774.327381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5774.561774] Lustre: lustre-OST0000: new disk, initializing [ 5774.572504] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5774.586325] Lustre: 209625:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5781.173315] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5784.128700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5784.152985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5784.345530] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5790.494766] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5795.245749] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5795.495522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5800.331853] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5800.660737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5805.025241] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5805.025847] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5805.046308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5805.046316] Lustre: Skipped 9 previous similar messages [ 5805.066701] Lustre: Skipped 1 previous similar message [ 5805.105594] LustreError: Skipped 1 previous similar message [ 5808.834552] Lustre: server umount lustre-OST0000 complete [ 5818.938825] Lustre: server umount lustre-MDT0000 complete [ 5820.383927] LustreError: 207666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5820.405382] LustreError: 207666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 5822.612474] LustreError: 207651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246245 with bad export cookie 10406428575251250797 [ 5822.618846] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5822.634301] LustreError: 207651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5823.040227] Lustre: server umount lustre-MDT0001 complete [ 5831.490766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5836.427168] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5845.802743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5850.390722] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5857.499824] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5863.380392] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5869.806565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5870.179781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5870.182017] Lustre: Skipped 4 previous similar messages [ 5876.070767] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5882.470403] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5888.489654] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5888.753864] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5893.547523] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5893.850855] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5897.189197] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5897.202032] LustreError: Skipped 1 previous similar message [ 5902.178356] Lustre: server umount lustre-OST0000 complete [ 5909.538561] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5914.382523] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5942.399511] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5948.510078] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 5955.359752] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5962.534167] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5963.172211] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.193218] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5963.196335] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5963.226820] Lustre: Skipped 1 previous similar message [ 5968.352129] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5968.360895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5968.370864] Lustre: Skipped 5 previous similar messages [ 5968.395990] Lustre: Skipped 5 previous similar messages [ 5973.316938] Lustre: server umount lustre-OST0000 complete [ 5991.391860] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5991.545707] Lustre: server umount lustre-MDT0000 complete [ 5993.952101] LustreError: 213085:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5993.990668] LustreError: 213085:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5995.230940] LustreError: 211824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246417 with bad export cookie 10406428575251251679 [ 5995.241355] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5995.712794] Lustre: server umount lustre-MDT0001 complete [ 6003.518963] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 16:27:04 (1782246424) [ 6009.671060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.973943] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6021.454495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6027.053656] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6036.978485] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6045.976268] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6056.104771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.709134] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6070.716541] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6074.980450] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6080.758765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 6085.950409] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6086.184313] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6096.519442] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.860668] Lustre: lustre-OST0001: new disk, initializing [ 6096.986835] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6097.006967] Lustre: 221216:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6103.302181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6103.354149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6103.540785] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6107.266677] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6120.154378] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6125.313237] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6125.698513] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6129.588158] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6129.856444] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6134.241548] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6134.251561] LustreError: Skipped 3 previous similar messages [ 6137.782356] Lustre: server umount lustre-OST0001 complete [ 6144.101449] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6144.302299] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6148.683541] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6149.005936] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6178.096326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6178.286224] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6178.289376] Lustre: Skipped 5 previous similar messages [ 6183.412973] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6183.424726] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6183.442234] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6183.453855] Lustre: Skipped 1 previous similar message [ 6185.293150] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6192.741634] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6194.214276] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6202.285744] Lustre: server umount lustre-OST0001 complete [ 6224.367056] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6224.377131] Lustre: Skipped 13 previous similar messages [ 6233.056254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6233.302596] Lustre: server umount lustre-OST0000 complete [ 6237.155112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6237.181807] Lustre: Skipped 8 previous similar messages [ 6243.170395] Lustre: server umount lustre-MDT0000 complete [ 6246.914288] LustreError: 217791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246669 with bad export cookie 10406428575251252645 [ 6246.915650] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6246.923926] LustreError: 217791:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6261.216766] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6261.570142] Lustre: server umount lustre-MDT0001 complete [ 6285.394871] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 16:31:46 (1782246706) [ 6291.423666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6291.727875] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6291.766261] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6295.864509] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6303.475408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.585487] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6303.605858] Lustre: 226464:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6308.408943] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6316.280692] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6321.440579] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6327.496563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6327.663773] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6333.523314] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6339.949385] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6348.490868] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6348.546985] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6357.100987] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6367.023206] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6370.460605] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6376.081125] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6376.305759] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6380.625277] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6380.869268] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6385.945770] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff969389689800.ost_server_uuid 50 [ 6387.599880] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff969389689800.ost_server_uuid in IDLE state after 0 sec [ 6388.483711] Lustre: Permanently deactivating lustre-OST0001 [ 6388.509135] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6391.598198] Lustre: Permanently reactivating lustre-OST0001 [ 6397.933388] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6397.937828] LustreError: Skipped 3 previous similar messages [ 6400.416240] Lustre: server umount lustre-OST0001 complete [ 6403.058559] LustreError: 228305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6403.091767] LustreError: 228305:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 40 previous similar messages [ 6418.399096] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6418.878733] Lustre: server umount lustre-OST0000 complete [ 6437.343353] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6437.561275] Lustre: server umount lustre-MDT0000 complete [ 6441.146305] LustreError: 226456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782246863 with bad export cookie 10406428575251253933 [ 6441.147848] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6441.157744] LustreError: 226456:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6441.481881] Lustre: server umount lustre-MDT0001 complete [ 6462.270990] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 16:34:43 (1782246883) [ 6471.170668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6471.438046] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6471.455318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6471.464371] Lustre: Skipped 5 previous similar messages [ 6477.227546] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6485.346832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.413797] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6485.448300] Lustre: 233156:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6489.401520] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6495.766427] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6500.753893] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6506.459575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6506.650674] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6506.686698] Lustre: Permanently deactivating lustre-OST0000 [ 6506.700503] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6512.361602] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6514.712819] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6514.722186] LustreError: 188675:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000765469af x1868820052908032/t0(0) o8->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6518.108772] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6524.069195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6524.125535] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6529.460676] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6529.527979] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6535.865698] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6537.370698] Lustre: Permanently reactivating lustre-OST0000 [ 6537.387861] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6537.399947] Lustre: Skipped 2 previous similar messages [ 6539.058522] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6539.072873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6555.121877] Lustre: server umount lustre-OST0001 complete [ 6565.290827] Lustre: server umount lustre-OST0000 complete [ 6574.910460] Lustre: server umount lustre-MDT0000 complete [ 6578.475416] LustreError: 233149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782247001 with bad export cookie 10406428575251255578 [ 6578.479552] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6578.484218] LustreError: 233149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6578.818150] Lustre: server umount lustre-MDT0001 complete [ 6588.317260] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6591.136346] Key type lgssc unregistered [ 6591.365659] LNet: 238012:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6591.371160] LNetError: 238012:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6592.423768] LNet: Removed LNI 192.168.203.145@tcp [ 6593.212098] Key type .llcrypt unregistered [ 6593.216726] Key type ._llcrypt unregistered [ 6614.824265] Key type ._llcrypt registered [ 6614.825676] Key type .llcrypt registered [ 6614.881538] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6615.741564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6615.845749] alg: No test for adler32 (adler32-zlib) [ 6616.859249] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 6617.122295] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 6618.807183] Key type lgssc registered [ 6619.830905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6630.656320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6630.680168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6632.240795] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6637.059400] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6641.650354] LustreError: 239480:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6641.670037] LustreError: 239480:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6645.599603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6645.674098] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6645.701276] Lustre: 239470:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6645.710749] Lustre: Permanently deactivating lustre-MDT0001 [ 6645.721915] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6645.956901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6650.486352] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6651.926634] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6656.740758] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6662.031175] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6669.753521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6670.174531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6671.217175] LustreError: 241324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6671.267775] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6676.395798] LustreError: 241324:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6676.412026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6676.431825] LustreError: 241324:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6677.862303] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6681.590910] LustreError: 241325:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6681.612841] LustreError: 241325:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6685.213373] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6686.698476] LustreError: 241373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6686.720454] LustreError: 241373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6692.253668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6692.529489] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6697.971466] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6697.989591] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6698.945404] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6705.303797] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6707.819990] Lustre: Permanently reactivating lustre-MDT0001 [ 6707.844835] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6707.859909] Lustre: Skipped 2 previous similar messages [ 6711.554196] Lustre: lustre-MDT0001: Client bf9e3a44-148a-484f-b9db-cab3d3a4ae02 (at 192.168.203.45@tcp) reconnecting [ 6714.600367] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6714.614177] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 6714.634251] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 6714.663328] LustreError: 243100:0:(llog_osd.c:2259:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 6714.666236] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6714.684266] LustreError: 243100:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 6714.699599] LustreError: 243100:0:(obd_config.c:2103:class_config_llog_handler()) MGC192.168.203.145@tcp: cfg command failed: rc = -108 [ 6714.712830] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 6714.712830] [ 6714.724647] LustreError: 239472:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 6719.285329] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6721.453196] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6728.378903] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6728.591239] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6733.275129] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6733.436513] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6736.102656] Lustre: Permanently deactivating lustre-MDT0001 [ 6736.122510] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6736.132306] Lustre: Skipped 2 previous similar messages [ 6745.213788] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6753.766510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6753.785351] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6753.805062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6759.318523] Lustre: server umount lustre-MDT0000 complete [ 6759.395630] LustreError: 239476:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6759.421138] LustreError: 239476:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6762.979493] LustreError: 240232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782247185 with bad export cookie 3925021434891529814 [ 6762.994515] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6762.998705] LustreError: 240232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6763.449693] Lustre: server umount lustre-MDT0001 complete [ 6773.195726] Lustre: server umount lustre-OST0000 complete [ 6784.107756] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 16:40:04 (1782247204) [ 6793.550106] Lustre: server umount lustre-OST0001 complete [ 6800.292893] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 6810.018755] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 6845.419764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6845.681458] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6845.688417] Lustre: Skipped 2 previous similar messages [ 6845.733441] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6845.853953] Lustre: lustre-MDT0000: new disk, initializing [ 6845.983471] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6846.005135] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6850.588298] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6860.245425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.340845] Lustre: 247761:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6860.415074] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6860.418673] Lustre: Skipped 1 previous similar message [ 6860.482948] Lustre: lustre-MDT0001: new disk, initializing [ 6860.553671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6860.585738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6860.602477] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6865.149481] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6875.107510] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6880.849944] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6887.429180] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6887.707519] Lustre: lustre-OST0000: new disk, initializing [ 6887.712218] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6887.726516] Lustre: 249726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6887.858140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6894.577900] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6896.708972] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6896.720222] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6896.848426] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6903.267411] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6905.121445] LustreError: 247768:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6912.497682] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6912.628681] Lustre: lustre-OST0001: new disk, initializing [ 6912.632051] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6912.639205] Lustre: 250841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6915.196620] LustreError: 247768:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6915.238361] LustreError: 250445:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6918.227807] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 6921.285895] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6921.297919] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6921.386195] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6925.255192] LustreError: 250445:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6926.365505] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6927.824978] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6929.905070] Lustre: lustre-OST0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 6929.911428] Lustre: Skipped 4 previous similar messages [ 6931.949253] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6931.950854] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.964400] LustreError: Skipped 1 previous similar message [ 6931.965262] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6931.973067] Lustre: Skipped 4 previous similar messages [ 6934.997296] Lustre: lustre-OST0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 6935.014468] Lustre: Skipped 1 previous similar message [ 6935.739262] Lustre: server umount lustre-OST0001 complete [ 6937.063511] LustreError: 249710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6937.091666] LustreError: 249710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6953.440359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6953.442972] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6953.444313] LustreError: Skipped 1 previous similar message [ 6953.460405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6959.014477] Lustre: server umount lustre-OST0000 complete [ 6963.172297] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6963.180577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6963.194461] Lustre: Skipped 1 previous similar message [ 6963.208065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6963.216613] Lustre: Skipped 3 previous similar messages [ 6969.076440] Lustre: server umount lustre-MDT0000 complete [ 6972.615741] LustreError: 247752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782247395 with bad export cookie 3925021434891532635 [ 6972.616600] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6972.626157] LustreError: 247752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6979.297450] Lustre: server umount lustre-MDT0001 complete [ 6989.599109] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 6992.598922] Key type lgssc unregistered [ 6993.020091] LNet: 252876:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6993.033392] LNetError: 252876:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6993.059063] LNet: Removed LNI 192.168.203.145@tcp [ 6994.476753] Key type .llcrypt unregistered [ 6994.480406] Key type ._llcrypt unregistered [ 7035.984200] Key type ._llcrypt registered [ 7035.986245] Key type .llcrypt registered [ 7036.108492] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7037.242937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7037.252869] alg: No test for adler32 (adler32-zlib) [ 7038.407949] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7038.754024] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7040.378671] Key type lgssc registered [ 7041.435192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7048.583144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7048.603793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7049.856330] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7049.877290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7050.108458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7054.485485] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7063.087676] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7063.236121] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7063.277167] Lustre: 255101:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7063.605497] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7067.969181] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7075.858534] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7081.789707] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7088.299411] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7088.474603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7088.641467] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7095.606334] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7096.442188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7103.467614] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7156.200725] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7156.222372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7161.320229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7161.326436] Lustre: Skipped 2 previous similar messages [ 7161.662820] Lustre: server umount lustre-OST0000 complete [ 7165.591160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7177.931523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7178.326777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7184.373541] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7186.539277] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7186.558590] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7186.562476] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7186.578380] Lustre: Skipped 1 previous similar message [ 7191.476270] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7232.998450] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7232.998879] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7233.005399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7233.031636] Lustre: Skipped 2 previous similar messages [ 7235.902216] Lustre: server umount lustre-OST0000 complete [ 7242.720634] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7242.724440] LustreError: Skipped 1 previous similar message [ 7242.726872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7242.735718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7242.739421] Lustre: Skipped 1 previous similar message [ 7245.516391] Lustre: server umount lustre-MDT0000 complete [ 7248.540534] LustreError: 255841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782247671 with bad export cookie 16919094178637338139 [ 7248.546777] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7248.552918] LustreError: 255841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7248.860987] Lustre: server umount lustre-MDT0001 complete [ 7259.603229] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7262.694593] Key type lgssc unregistered [ 7262.975953] LNet: 260520:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7262.986698] LNetError: 260520:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7263.001934] LNet: Removed LNI 192.168.203.145@tcp [ 7263.927323] Key type .llcrypt unregistered [ 7263.929749] Key type ._llcrypt unregistered [ 7288.829342] Key type ._llcrypt registered [ 7288.834036] Key type .llcrypt registered [ 7288.941813] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7290.301687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7290.379552] alg: No test for adler32 (adler32-zlib) [ 7291.681410] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7291.981776] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7293.639383] Key type lgssc registered [ 7294.696076] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7302.171245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7302.221544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7303.646551] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7307.775412] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7313.891023] LustreError: 261936:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7313.913053] LustreError: 261936:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7315.527182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7315.956312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7319.506892] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7326.000685] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7331.075835] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7336.962302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7337.303801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7343.551484] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7348.588517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7351.110524] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7374.309133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7374.315509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.378484] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7374.388109] Lustre: Skipped 1 previous similar message [ 7378.997595] Lustre: server umount lustre-OST0000 complete [ 7384.551033] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7384.555083] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7384.564875] Lustre: Skipped 2 previous similar messages [ 7384.570439] Lustre: Skipped 1 previous similar message [ 7389.579681] Lustre: server umount lustre-MDT0000 complete [ 7389.671222] LustreError: 262665:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7393.981358] LustreError: 261917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782247816 with bad export cookie 5229280378590151865 [ 7393.996922] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7394.010868] LustreError: 261917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7394.416246] Lustre: server umount lustre-MDT0001 complete [ 7404.722728] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7407.204430] Key type lgssc unregistered [ 7407.451262] LNet: 266245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7407.461254] LNetError: 266245:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7407.476951] LNet: Removed LNI 192.168.203.145@tcp [ 7408.379737] Key type .llcrypt unregistered [ 7408.382824] Key type ._llcrypt unregistered [ 7427.476199] Key type ._llcrypt registered [ 7427.483087] Key type .llcrypt registered [ 7427.634673] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7428.841063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7428.915576] alg: No test for adler32 (adler32-zlib) [ 7430.088634] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7430.357212] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7432.080086] Key type lgssc registered [ 7433.141794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7441.244583] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7441.286491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7442.994500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7448.218237] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7453.157099] LustreError: 267474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7453.186295] LustreError: 267474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7457.172322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7457.723530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7462.895774] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7471.467815] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7477.033863] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7483.749371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7484.134233] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7488.249990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7490.882379] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7498.664695] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7514.081058] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7514.089148] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7514.108611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7518.182374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7518.183923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7518.195349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7518.223303] Lustre: Skipped 1 previous similar message [ 7519.444052] Lustre: server umount lustre-OST0000 complete [ 7523.298559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.312121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7523.322618] Lustre: Skipped 1 previous similar message [ 7523.336735] Lustre: Skipped 1 previous similar message [ 7528.421280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7528.428909] Lustre: Skipped 1 previous similar message [ 7533.537557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7533.545386] Lustre: Skipped 1 previous similar message [ 7537.119931] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7537.292762] Lustre: server umount lustre-MDT0000 complete [ 7538.656786] LustreError: 267474:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7538.678363] LustreError: 267474:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7540.701455] LustreError: 267454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782247963 with bad export cookie 15153237161530871461 [ 7540.713190] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7541.024591] Lustre: server umount lustre-MDT0001 complete [ 7550.089725] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7552.637947] Key type lgssc unregistered [ 7552.894090] LNet: 271152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7552.909217] LNetError: 271152:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7553.974593] LNet: Removed LNI 192.168.203.145@tcp [ 7554.924804] Key type .llcrypt unregistered [ 7554.927049] Key type ._llcrypt unregistered [ 7577.090575] Key type ._llcrypt registered [ 7577.095206] Key type .llcrypt registered [ 7577.209680] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7578.632230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7578.668395] alg: No test for adler32 (adler32-zlib) [ 7579.782241] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7579.985385] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7581.663164] Key type lgssc registered [ 7582.839800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7590.466243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7590.497607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7591.951396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7596.353726] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7602.150571] LustreError: 272568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7602.176842] LustreError: 272568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7604.049637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7604.435256] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7608.657249] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7615.909166] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7621.532643] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7628.541148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7628.917870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7634.927181] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7637.111274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7642.068834] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7666.146379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7666.158236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.176879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.207693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7668.213412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7668.245646] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7671.377292] Lustre: server umount lustre-OST0000 complete [ 7676.385590] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7676.396976] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7676.416956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7676.431147] Lustre: Skipped 1 previous similar message [ 7678.435182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7678.439036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7678.462267] Lustre: Skipped 1 previous similar message [ 7681.151447] Lustre: server umount lustre-MDT0000 complete [ 7683.560137] LustreError: 272568:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7683.587982] LustreError: 272568:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7684.505254] LustreError: 272549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248107 with bad export cookie 6092990991958341586 [ 7684.518251] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7684.523671] LustreError: 272549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7684.916686] Lustre: server umount lustre-MDT0001 complete [ 7694.337630] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7696.922526] Key type lgssc unregistered [ 7697.317470] LNet: 276874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7697.353481] LNetError: 276874:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7697.376608] LNet: Removed LNI 192.168.203.145@tcp [ 7698.440563] Key type .llcrypt unregistered [ 7698.447229] Key type ._llcrypt unregistered [ 7716.410566] Key type ._llcrypt registered [ 7716.412962] Key type .llcrypt registered [ 7716.555879] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7718.076575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7718.125116] alg: No test for adler32 (adler32-zlib) [ 7719.168821] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7719.447644] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7721.151538] Key type lgssc registered [ 7722.057859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7729.736838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7729.761168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7731.405294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7735.814396] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7741.926434] LustreError: 278112:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7741.945536] LustreError: 278112:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7744.326895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7744.739282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7750.616299] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7758.561950] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7765.428257] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7773.457165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.830946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7776.897434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7780.784442] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7788.694976] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7800.806601] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7800.811391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7800.813253] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7800.825046] Lustre: Skipped 1 previous similar message [ 7805.930111] LustreError: 280628:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.097938] Lustre: server umount lustre-OST0000 complete [ 7811.048808] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7811.071873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7811.083963] Lustre: Skipped 2 previous similar messages [ 7816.174452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.178830] Lustre: Skipped 1 previous similar message [ 7821.282752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7821.291656] Lustre: Skipped 1 previous similar message [ 7824.351124] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7824.540819] Lustre: server umount lustre-MDT0000 complete [ 7826.401240] LustreError: 278106:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7826.420611] LustreError: 278106:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7828.273695] LustreError: 278077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248251 with bad export cookie 14292425857223100504 [ 7828.286849] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7828.289810] LustreError: 278077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7828.714947] Lustre: server umount lustre-MDT0001 complete [ 7840.525169] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 7843.613505] Key type lgssc unregistered [ 7843.952206] LNet: 281815:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7843.966260] LNetError: 281815:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7843.992663] LNet: Removed LNI 192.168.203.145@tcp [ 7845.036749] Key type .llcrypt unregistered [ 7845.044524] Key type ._llcrypt unregistered [ 7857.196538] Key type ._llcrypt registered [ 7857.200355] Key type .llcrypt registered [ 7857.330729] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7874.613218] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 7889.720907] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 7890.717290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7890.738827] alg: No test for adler32 (adler32-zlib) [ 7891.804302] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 7892.151787] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 7893.840797] Key type lgssc registered [ 7894.979708] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7928.050210] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7928.085564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7929.352268] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7929.404849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7929.489203] Lustre: lustre-MDT0000: new disk, initializing [ 7929.591098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7929.611054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7934.484221] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7945.311499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.464913] Lustre: 285364:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7945.535453] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7945.543436] Lustre: Skipped 1 previous similar message [ 7945.666373] Lustre: lustre-MDT0001: new disk, initializing [ 7945.774272] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7945.822132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7945.834794] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7950.773831] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7960.184736] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7965.781127] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7973.319237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7973.532420] Lustre: lustre-OST0000: new disk, initializing [ 7973.535734] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7973.541405] Lustre: 287332:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7973.645843] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7980.883075] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 7983.129668] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7983.146890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7983.206438] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7990.627980] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7996.052657] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7996.234263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8001.411755] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8001.679932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8003.569236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8003.579402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8003.593669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8008.675902] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8008.675916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8008.691186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8008.708170] Lustre: Skipped 1 previous similar message [ 8009.953224] Lustre: server umount lustre-OST0000 complete [ 8017.377204] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8017.389212] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8017.402654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8020.244382] Lustre: server umount lustre-MDT0000 complete [ 8024.342292] LustreError: 287342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248447 with bad export cookie 18301738960810434122 [ 8024.348451] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8024.353713] LustreError: 287342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8024.764828] Lustre: server umount lustre-MDT0001 complete [ 8034.418294] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 17:00:55 (1782248455) [ 8041.756360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8042.335398] LustreError: 289761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8042.363734] LustreError: 289761:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 8042.448752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.472528] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8047.585064] LustreError: 289762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8052.706844] LustreError: 289761:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8055.499054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8055.776984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8059.804766] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8067.439882] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8074.035142] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8081.708527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8082.182369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.439717] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8097.361865] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8111.082713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8111.088401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8111.092322] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8111.092329] Lustre: Skipped 2 previous similar messages [ 8111.134109] Lustre: Skipped 2 previous similar messages [ 8116.203329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8116.207284] Lustre: Skipped 1 previous similar message [ 8116.808526] Lustre: server umount lustre-OST0000 complete [ 8121.318510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8121.342579] Lustre: Skipped 1 previous similar message [ 8126.434444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8126.445011] Lustre: Skipped 4 previous similar messages [ 8126.732165] Lustre: server umount lustre-MDT0000 complete [ 8130.318919] LustreError: 289743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248553 with bad export cookie 18301738960810434997 [ 8130.324824] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8130.325413] LustreError: 289743:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8130.868646] Lustre: server umount lustre-MDT0001 complete [ 8141.406310] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8144.368737] Key type lgssc unregistered [ 8144.618900] LNet: 293320:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8144.627379] LNetError: 293320:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8144.651888] LNet: Removed LNI 192.168.203.145@tcp [ 8145.453741] Key type .llcrypt unregistered [ 8145.457315] Key type ._llcrypt unregistered [ 8171.486846] Key type ._llcrypt registered [ 8171.491463] Key type .llcrypt registered [ 8171.601986] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8188.451545] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8189.586963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8189.615439] alg: No test for adler32 (adler32-zlib) [ 8190.950458] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 8191.384916] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8193.111226] Key type lgssc registered [ 8194.449859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8246.914149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8246.958393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8248.263127] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8248.291124] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8248.429879] Lustre: lustre-MDT0000: new disk, initializing [ 8248.576784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.615080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8253.374080] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8264.470575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8264.645194] Lustre: 297147:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8264.708431] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8264.715344] Lustre: Skipped 1 previous similar message [ 8264.834370] Lustre: lustre-MDT0001: new disk, initializing [ 8264.985132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8265.066119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8265.079912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8270.072202] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8280.434982] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8287.231619] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8294.849827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8295.201336] Lustre: lustre-OST2710: new disk, initializing [ 8295.214637] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8295.222919] Lustre: 299114:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8295.369110] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8301.692049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8301.716125] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8301.847160] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8302.625748] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8313.114549] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8321.323817] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8321.569992] Lustre: lustre-OST03e8: new disk, initializing [ 8321.573137] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8321.582493] Lustre: 300181:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8321.736575] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8328.993342] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8330.335799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8330.344970] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8330.449960] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8337.784855] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8346.688203] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8347.004530] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8352.954029] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8353.262698] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8358.823137] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8359.115742] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8364.474263] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8364.878792] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8371.686276] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8371.694522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8371.697783] Lustre: Skipped 1 previous similar message [ 8371.724492] Lustre: Skipped 3 previous similar messages [ 8372.390610] Lustre: server umount lustre-MDT0000 complete [ 8376.728543] LustreError: 297139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248799 with bad export cookie 9972295798346275845 [ 8376.737167] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8376.741154] LustreError: 297139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8376.803518] Lustre: lustre-MDT0001-lwp-OST2710: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8376.817878] Lustre: Skipped 2 previous similar messages [ 8376.828034] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8376.846476] Lustre: Skipped 1 previous similar message [ 8377.438258] Lustre: server umount lustre-MDT0001 complete [ 8383.776575] Lustre: server umount lustre-OST2710 complete [ 8388.136146] Lustre: server umount lustre-OST03e8 complete [ 8406.800931] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 8416.874616] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8451.327137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8451.577311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8451.610291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8451.719956] Lustre: lustre-MDT0000: new disk, initializing [ 8451.815849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.845675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8455.716617] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8466.344153] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8466.464871] Lustre: 305429:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8466.518818] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8466.525170] Lustre: Skipped 1 previous similar message [ 8466.606632] Lustre: lustre-MDT0001: new disk, initializing [ 8466.809912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8466.861786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8466.866912] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8472.650918] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8480.992640] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8485.978284] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8491.929756] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8492.297749] Lustre: lustre-OST0000: new disk, initializing [ 8492.303554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8492.315240] Lustre: 307398:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8492.407964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8498.912949] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8501.323200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8501.333839] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8501.434304] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8507.473704] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8512.016277] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8512.300202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8516.938166] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8517.211262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8522.214241] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8522.216359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8522.225325] LustreError: Skipped 1 previous similar message [ 8522.240390] Lustre: Skipped 1 previous similar message [ 8522.242071] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8525.441703] Lustre: server umount lustre-OST0000 complete [ 8532.453536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8532.485044] Lustre: Skipped 1 previous similar message [ 8532.496412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8532.510129] Lustre: Skipped 1 previous similar message [ 8535.767207] Lustre: server umount lustre-MDT0000 complete [ 8537.569221] LustreError: 305441:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.588817] LustreError: 305441:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8539.194340] LustreError: 305422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782248961 with bad export cookie 9972295798346277532 [ 8539.196302] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8539.212189] LustreError: 305422:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8539.793804] Lustre: server umount lustre-MDT0001 complete [ 8547.830917] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 17:09:28 (1782248968) [ 8549.334554] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8550.940862] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 17:09:31 (1782248971) [ 8554.740327] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8580.737887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.084549] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8581.110869] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8581.348890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8585.658979] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8593.022146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8593.118582] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8593.140447] Lustre: 311255:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8597.486229] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8618.165025] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8618.420512] Lustre: Denying initial registration attempt from nid 192.168.203.145@tcp, specified as failover [ 8618.434899] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8618.454861] LustreError: 312707:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8618.468445] LustreError: 312707:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8618.475484] LustreError: 312707:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8618.546411] Lustre: server umount lustre-OST0000 complete [ 8618.551672] LustreError: 312707:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8624.611163] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8624.626028] Lustre: Skipped 1 previous similar message [ 8624.631333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8629.217711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8630.905987] Lustre: server umount lustre-MDT0000 complete [ 8633.984270] LustreError: 311245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249056 with bad export cookie 9972295798346278414 [ 8633.989747] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8633.994431] LustreError: 311245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8634.376982] Lustre: server umount lustre-MDT0001 complete [ 8643.631704] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8646.507631] Key type lgssc unregistered [ 8646.787137] LNet: 313827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8646.800983] LNetError: 313827:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8646.823579] LNet: Removed LNI 192.168.203.145@tcp [ 8648.126792] Key type .llcrypt unregistered [ 8648.132949] Key type ._llcrypt unregistered [ 8661.706756] Key type ._llcrypt registered [ 8661.710776] Key type .llcrypt registered [ 8661.834725] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8662.725524] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8662.746074] alg: No test for adler32 (adler32-zlib) [ 8663.748096] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 8663.899910] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8665.583305] Key type lgssc registered [ 8666.830026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8710.600261] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8718.652958] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8718.682570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.866823] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8719.905860] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8720.080241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8724.342910] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8732.571256] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8732.719973] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8732.754138] Lustre: 316594:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8733.021562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8737.581461] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8744.262184] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8749.255506] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8768.701717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8769.009664] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8769.239146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8775.351342] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8782.146243] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8784.355066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8784.359335] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8784.370094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8784.380537] Lustre: Skipped 1 previous similar message [ 8789.474121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8789.481655] Lustre: Skipped 2 previous similar messages [ 8790.160610] Lustre: server umount lustre-OST0000 complete [ 8794.599432] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.603448] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8794.603873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8794.614584] Lustre: Skipped 1 previous similar message [ 8799.681851] Lustre: server umount lustre-MDT0000 complete [ 8802.845882] LustreError: 317333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249225 with bad export cookie 4157092067494678092 [ 8802.850444] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8802.860214] LustreError: 317333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8803.281898] Lustre: server umount lustre-MDT0001 complete [ 8813.892383] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 8816.972408] Key type lgssc unregistered [ 8817.311052] LNet: 320164:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8817.322433] LNetError: 320164:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8817.345629] LNet: Removed LNI 192.168.203.145@tcp [ 8818.257549] Key type .llcrypt unregistered [ 8818.260799] Key type ._llcrypt unregistered [ 8840.515878] Key type ._llcrypt registered [ 8840.518429] Key type .llcrypt registered [ 8840.658906] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 8841.867296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8841.901087] alg: No test for adler32 (adler32-zlib) [ 8842.943579] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 8843.139614] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 8844.823224] Key type lgssc registered [ 8845.736794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8852.784345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8852.799365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8854.185702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8858.263497] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8863.721762] LustreError: 321582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8863.736721] LustreError: 321582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8865.894354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.216945] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8869.905358] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8876.614662] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8881.544143] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8888.939255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8889.154171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8893.242839] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8898.399344] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8905.195824] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [ 8907.234820] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8907.241563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8907.258481] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8908.777285] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8908.777962] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8908.782545] Lustre: Skipped 1 previous similar message [ 8908.802128] Lustre: Skipped 1 previous similar message [ 8911.210376] Lustre: server umount lustre-MDT0000 complete [ 8913.894266] LustreError: 321578:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8913.917153] LustreError: 321578:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8914.469865] LustreError: 324166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249337 with bad export cookie 161406953705827885 [ 8914.473043] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8914.478164] LustreError: 324166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8915.397743] LustreError: 321577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8915.424371] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 8915.432597] Lustre: Skipped 2 previous similar messages [ 8919.009214] LustreError: 321577:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8919.019791] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.513897] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [ 8920.527268] Lustre: Skipped 1 previous similar message [ 8920.956911] Lustre: server umount lustre-MDT0001 complete [ 8926.602439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8933.900758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8940.025199] LustreError: 323415:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8940.058620] LustreError: 323415:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8940.144546] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8940.244120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8946.302404] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8948.193107] LustreError: 323416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8948.203060] LustreError: 323416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8949.221734] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8955.990361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8956.238057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8958.329592] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8961.053675] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 8968.505032] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8970.074413] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8975.813961] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8977.433140] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8981.985427] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8981.992912] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8982.013185] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8987.952809] Lustre: server umount lustre-OST0000 complete [ 8992.230349] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8992.242888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8992.258299] Lustre: Skipped 1 previous similar message [ 8997.524750] Lustre: server umount lustre-MDT0000 complete [ 8999.397429] LustreError: 325177:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8999.422128] LustreError: 325177:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9000.955864] LustreError: 321564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249423 with bad export cookie 161406953705835564 [ 9000.956306] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9000.964644] LustreError: 321564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9001.271450] Lustre: server umount lustre-MDT0001 complete [ 9011.725735] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9014.542861] Key type lgssc unregistered [ 9014.852617] LNet: 327851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9014.863880] LNetError: 327851:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9014.884221] LNet: Removed LNI 192.168.203.145@tcp [ 9015.688540] Key type .llcrypt unregistered [ 9015.693088] Key type ._llcrypt unregistered [ 9037.588570] Key type ._llcrypt registered [ 9037.591518] Key type .llcrypt registered [ 9037.707544] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9038.955356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9038.978069] alg: No test for adler32 (adler32-zlib) [ 9040.057100] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 9040.282297] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9041.935160] Key type lgssc registered [ 9042.932582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9051.038450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9051.083916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9052.762263] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9058.042205] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9062.883311] LustreError: 329262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9062.897534] LustreError: 329262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9066.139777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.488042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9069.883518] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9075.779526] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9081.113733] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9086.949138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9086.968710] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9086.983503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9087.971519] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9087.973228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9089.639781] Lustre: server umount lustre-MDT0000 complete [ 9092.128885] LustreError: 329262:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.152068] LustreError: 329262:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9092.552523] LustreError: 329241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249515 with bad export cookie 11932423563937608424 [ 9092.565749] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9092.877137] Lustre: server umount lustre-MDT0001 complete [ 9105.365528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9105.782727] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9105.827572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9106.051632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9110.207579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9117.410422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9117.471619] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9117.493620] Lustre: 331849:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9117.728756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9122.111818] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9128.762047] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9135.355924] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9142.634703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9142.832396] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9142.843213] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9142.849984] LustreError: 333734:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9142.855214] LustreError: 333734:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9142.858167] LustreError: 333734:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9142.919080] Lustre: server umount lustre-OST0000 complete [ 9142.922318] LustreError: 333734:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9153.218655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9153.402327] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9153.529494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9159.388500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9159.595701] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9166.588145] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9173.932707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9174.097942] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9174.212619] Lustre: lustre-OST0001: new disk, initializing [ 9174.221119] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9174.234484] Lustre: 335277:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9174.391887] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9179.754449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9179.763285] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9179.887710] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9181.713498] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9188.943655] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9195.487861] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9195.498356] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9195.501902] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9195.516484] Lustre: Skipped 1 previous similar message [ 9195.529749] Lustre: Skipped 2 previous similar messages [ 9197.552879] Lustre: server umount lustre-OST0001 complete [ 9200.610051] LustreError: 335563:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9200.632440] LustreError: 335563:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9205.732956] LustreError: 334202:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9205.733385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9205.733629] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9205.733634] LustreError: Skipped 1 previous similar message [ 9205.744916] LustreError: 334202:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9205.774640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9205.778523] Lustre: Skipped 1 previous similar message [ 9206.973232] Lustre: server umount lustre-OST0000 complete [ 9210.849638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9210.854708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9210.879089] Lustre: Skipped 2 previous similar messages [ 9216.776784] Lustre: server umount lustre-MDT0000 complete [ 9220.005916] LustreError: 331841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249642 with bad export cookie 11932423563937608977 [ 9220.014777] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9220.016694] LustreError: 331841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 9220.338185] Lustre: server umount lustre-MDT0001 complete [ 9229.523468] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9232.251259] Key type lgssc unregistered [ 9232.569591] LNet: 337235:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9232.574774] LNetError: 337235:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9232.586750] LNet: Removed LNI 192.168.203.145@tcp [ 9233.741498] Key type .llcrypt unregistered [ 9233.744851] Key type ._llcrypt unregistered [ 9296.327874] Key type ._llcrypt registered [ 9296.330488] Key type .llcrypt registered [ 9296.436694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [ 9308.938551] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9309.807293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9309.824623] alg: No test for adler32 (adler32-zlib) [ 9310.996331] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 9311.223417] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9312.879386] Key type lgssc registered [ 9314.256662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9349.429251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9349.457976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9350.756087] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9350.781424] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9350.878168] Lustre: lustre-MDT0000: new disk, initializing [ 9350.982908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9351.000954] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9355.188761] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9365.035777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9365.142663] Lustre: 342148:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9365.210114] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9365.215938] Lustre: Skipped 1 previous similar message [ 9365.291221] Lustre: lustre-MDT0001: new disk, initializing [ 9365.390802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9365.431283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9365.447249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9369.341444] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9378.801568] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9384.359347] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9391.702882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9391.892272] Lustre: lustre-OST0000: new disk, initializing [ 9391.895715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9391.908455] Lustre: 344113:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9391.979837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9398.395045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9398.410531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9398.517183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9398.838915] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9406.813317] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9411.169200] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9411.390637] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9415.323264] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9415.546658] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9419.247988] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9419.256422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9419.279530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9423.803739] Lustre: server umount lustre-OST0000 complete [ 9429.472761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9429.475126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9429.479892] Lustre: Skipped 2 previous similar messages [ 9429.484438] Lustre: Skipped 1 previous similar message [ 9434.595918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9434.598867] Lustre: Skipped 2 previous similar messages [ 9439.713686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9439.716871] Lustre: Skipped 1 previous similar message [ 9441.251660] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9441.447117] Lustre: server umount lustre-MDT0000 complete [ 9444.675150] LustreError: 342138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782249867 with bad export cookie 7732411209314482802 [ 9444.679764] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9444.685705] LustreError: 342138:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9444.968258] Lustre: server umount lustre-MDT0001 complete [ 9452.803469] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 17:24:33 (1782249873) [ 9459.717304] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 17:24:40 (1782249880) [ 9466.716351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9467.130962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9471.223163] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9472.482050] LustreError: 346873:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9472.497945] LustreError: 346873:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9476.579896] LustreError: 346874:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9478.374210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9478.734830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9482.506902] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9489.291104] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9494.527947] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9499.752962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9500.034921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9505.708089] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9512.266960] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9520.088635] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9521.773782] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9523.359245] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9524.772226] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9526.609882] Lustre: Failing over lustre-MDT0000 [ 9526.754693] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9526.769910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9526.789520] Lustre: Skipped 1 previous similar message [ 9526.997648] Lustre: server umount lustre-MDT0000 complete [ 9530.332818] LustreError: 346869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9535.440828] LustreError: 349432:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9535.467800] LustreError: 349432:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9540.561043] LustreError: 346868:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9540.581160] LustreError: 346868:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9544.343687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9544.431584] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9544.645382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.687884] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9545.671410] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9548.272848] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9549.809565] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9549.853934] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9549.896141] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9557.016033] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9558.580821] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9561.835393] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9563.729348] Lustre: Failing over lustre-MDT0000 [ 9563.901150] LustreError: 350836:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000beecc4bc) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 9565.155127] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9565.155723] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9565.160549] LustreError: Skipped 1 previous similar message [ 9565.170490] Lustre: Skipped 1 previous similar message [ 9565.171674] LustreError: 346869:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9565.209612] LustreError: 346869:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 9566.013781] Lustre: server umount lustre-MDT0000 complete [ 9581.024807] Lustre: 340371:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782249987/real 1782249987] req@000000005b93d252 x1868824296115328/t0(0) o400->MGC192.168.203.145@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1782250003 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9581.060648] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9581.516791] LustreError: 348978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9581.541305] LustreError: 348978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 15 previous similar messages [ 9584.549819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.286828] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6b4f0b06dc704507 [ 9591.299863] Lustre: MGC192.168.203.145@tcp: Connection restored to 0@lo (at 0@lo) [ 9591.318592] Lustre: Skipped 2 previous similar messages [ 9591.599640] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9591.664656] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9592.782750] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9596.544213] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9596.924961] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9596.972695] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 9597.011606] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9606.771728] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9608.792535] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9617.377800] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9617.382062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9617.395112] Lustre: Skipped 4 previous similar messages [ 9617.403475] Lustre: Skipped 3 previous similar messages [ 9621.554758] Lustre: server umount lustre-MDT0000 complete [ 9622.506761] LustreError: 347600:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9622.535874] LustreError: 347600:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 9625.891199] LustreError: 346853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250048 with bad export cookie 7732411209314485511 [ 9625.894418] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9625.907022] LustreError: 346853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9626.331281] Lustre: server umount lustre-MDT0001 complete [ 9640.928224] Lustre: server umount lustre-OST0000 complete [ 9657.424146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9657.929804] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9662.363853] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9669.917439] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9674.167822] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9681.584175] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9686.612744] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9692.958161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9700.017170] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9702.575340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9707.501813] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9713.126208] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9713.140019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9713.148873] Lustre: Skipped 1 previous similar message [ 9713.169455] Lustre: Skipped 1 previous similar message [ 9718.626741] Lustre: server umount lustre-OST0000 complete [ 9728.488449] LustreError: 353212:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9728.510802] LustreError: 353212:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9728.602893] Lustre: server umount lustre-MDT0000 complete [ 9732.355600] LustreError: 353194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250155 with bad export cookie 7732411209314486260 [ 9732.357766] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9732.362966] LustreError: 353194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9732.712462] Lustre: server umount lustre-MDT0001 complete [ 9744.427097] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [ 9747.844884] Key type lgssc unregistered [ 9748.185039] LNet: 356742:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9748.196247] LNetError: 356742:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9748.221884] LNet: Removed LNI 192.168.203.145@tcp [ 9749.271520] Key type .llcrypt unregistered [ 9749.272684] Key type ._llcrypt unregistered [ 9775.265252] Key type ._llcrypt registered [ 9775.267794] Key type .llcrypt registered [ 9775.360103] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [ 9776.188512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9776.269180] alg: No test for adler32 (adler32-zlib) [ 9777.355542] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [ 9777.591876] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [ 9779.271183] Key type lgssc registered [ 9780.738766] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9789.123983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9789.180025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9790.822574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9795.105912] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9801.186822] LustreError: 358269:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9801.217201] LustreError: 358269:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9803.267804] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9803.699996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9808.607444] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9815.811325] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9821.270031] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9827.771167] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9828.214174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9834.268344] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9835.379842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9841.609751] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9854.945663] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9854.958491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9854.984668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9855.161932] Lustre: server umount lustre-MDT0000 complete [ 9855.982449] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9855.982568] LustreError: 358268:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9856.035990] LustreError: 358268:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9859.301988] LustreError: 358250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250282 with bad export cookie 12304685368138753195 [ 9859.321821] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9859.332118] LustreError: 358250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9859.869862] Lustre: server umount lustre-MDT0001 complete [ 9875.167553] Lustre: server umount lustre-OST0000 complete [ 9887.311661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9887.692324] LustreError: 361977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9887.765357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9892.001444] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9892.849602] LustreError: 361978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.954984] LustreError: 361977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9900.583698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9901.102352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9905.552951] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9913.028752] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9918.697626] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9925.597739] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9925.966370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9932.123681] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [ 9933.249119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9939.183697] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9952.224537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9952.236950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.249697] Lustre: Skipped 1 previous similar message [ 9952.255814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9953.761945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9953.770423] Lustre: Skipped 1 previous similar message [ 9956.902292] Lustre: server umount lustre-MDT0000 complete [ 9958.881980] LustreError: 361972:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9958.900132] LustreError: 361972:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9960.792891] LustreError: 361957:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250383 with bad export cookie 12304685368138754308 [ 9960.804740] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9961.314106] Lustre: server umount lustre-MDT0001 complete [ 9974.554524] Lustre: server umount lustre-OST0000 complete [ 9991.190596] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9991.569154] LustreError: 365888:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9991.664994] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.051948] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10005.746865] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10010.336518] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10018.082657] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10023.277253] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10029.630853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10030.038345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10030.057083] Lustre: Skipped 1 previous similar message [10035.974079] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10037.253805] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10042.863179] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10049.946946] Lustre: server umount lustre-OST0000 complete [10057.185681] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10057.190974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10057.205727] Lustre: Skipped 2 previous similar messages [10057.211444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10062.818344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10062.821234] Lustre: Skipped 1 previous similar message [10065.947792] Lustre: server umount lustre-MDT0000 complete [10067.936920] Lustre: 357582:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782250474/real 1782250474] req@00000000ad8004e6 x1868824785805568/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782250490 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10067.938107] LustreError: 367128:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10067.965464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10067.965471] Lustre: Skipped 1 previous similar message [10067.991501] LustreError: 367128:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10068.959878] Lustre: 357579:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782250475/real 1782250475] req@00000000b544b2a2 x1868824785806336/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782250491 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10070.122157] LustreError: 365870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250492 with bad export cookie 12304685368138755330 [10070.137858] LustreError: 365870:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [10070.139366] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10070.492898] Lustre: server umount lustre-MDT0001 complete [10081.384686] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10084.627954] Key type lgssc unregistered [10085.024601] LNet: 369421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10085.035336] LNetError: 369421:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10085.081388] LNet: Removed LNI 192.168.203.145@tcp [10086.000128] Key type .llcrypt unregistered [10086.002920] Key type ._llcrypt unregistered [10109.427756] Key type ._llcrypt registered [10109.433988] Key type .llcrypt registered [10109.551902] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10110.355481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10110.376614] alg: No test for adler32 (adler32-zlib) [10111.399458] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [10111.588025] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10113.255836] Key type lgssc registered [10114.305085] Lustre: Echo OBD driver; http://www.lustre.org/ [10120.863512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10120.882569] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10120.892618] LustreError: 370864:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10120.908481] LustreError: 370864:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10120.916208] LustreError: 370864:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10120.926415] LustreError: 370864:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10120.946030] LustreError: 370864:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10126.696391] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10126.702557] LustreError: 371261:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10126.721651] LustreError: 371261:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10126.729588] LustreError: 371261:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10126.735917] LustreError: 371261:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10126.742637] LustreError: 371261:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [10142.978863] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10145.154638] Key type lgssc unregistered [10145.373389] LNet: 372263:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10145.380510] LNetError: 372263:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10145.400407] LNet: Removed LNI 192.168.203.145@tcp [10146.084333] Key type .llcrypt unregistered [10146.087550] Key type ._llcrypt unregistered [10160.776265] Key type ._llcrypt registered [10160.778284] Key type .llcrypt registered [10160.867845] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10172.679161] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10173.587209] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10173.615224] alg: No test for adler32 (adler32-zlib) [10174.662680] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [10174.936762] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10176.663169] Key type lgssc registered [10177.653647] Lustre: Echo OBD driver; http://www.lustre.org/ [10209.852105] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10209.880363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10211.129797] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10211.152344] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10211.242220] Lustre: lustre-MDT0000: new disk, initializing [10211.347747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10211.360688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10216.323586] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10225.820854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10225.961687] Lustre: 375398:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10226.026864] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10226.031133] Lustre: Skipped 1 previous similar message [10226.102861] Lustre: lustre-MDT0001: new disk, initializing [10226.217432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10226.268623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10226.284242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10230.501380] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10237.906678] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10241.952699] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10248.826521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10249.155287] Lustre: lustre-OST0000: new disk, initializing [10249.162805] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10249.172614] Lustre: 377368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10249.286849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10254.972679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10254.989498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10255.104793] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10255.487445] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10264.393264] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10268.948810] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10269.225442] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10273.779409] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10273.974118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10275.808646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10275.820990] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.838562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10275.847413] Lustre: Skipped 1 previous similar message [10280.937400] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10280.952798] Lustre: Skipped 1 previous similar message [10281.912517] Lustre: server umount lustre-OST0000 complete [10286.049982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10286.052813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10286.071988] Lustre: Skipped 2 previous similar messages [10286.091762] Lustre: Skipped 1 previous similar message [10291.171164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10291.181465] Lustre: Skipped 1 previous similar message [10291.595349] Lustre: server umount lustre-MDT0000 complete [10295.125894] LustreError: 376194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250717 with bad export cookie 1224588565813978450 [10295.138689] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10295.521428] Lustre: server umount lustre-MDT0001 complete [10302.981687] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10304.613476] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 17:38:45 (1782250725) [10310.397370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10310.970975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10316.010105] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10316.264887] LustreError: 379825:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10316.280644] LustreError: 379825:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10320.362711] LustreError: 379826:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10324.314809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10324.736511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10329.041603] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10336.548310] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10343.923922] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10351.498525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10352.007562] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10358.255226] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10364.954194] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10371.183332] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10371.365319] Lustre: lustre-OST0001: new disk, initializing [10371.368395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10371.373637] Lustre: 382660:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10371.459209] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10377.087582] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10379.864593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10379.882060] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10379.981456] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10385.830927] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [10390.510396] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10390.518372] LustreError: Skipped 1 previous similar message [10390.520702] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10390.525392] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10390.550816] Lustre: Skipped 1 previous similar message [10390.567231] Lustre: Skipped 1 previous similar message [10395.329970] Lustre: server umount lustre-OST0001 complete [10395.628272] LustreError: 383142:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10395.648121] LustreError: 383142:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10397.636237] LustreError: 381648:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.874055] LustreError: 381646:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10405.902498] LustreError: 381646:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10412.513774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10412.531554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10412.551709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10412.560888] Lustre: Skipped 1 previous similar message [10415.588535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10416.112295] LustreError: 381647:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10416.148176] LustreError: 381647:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10417.819906] Lustre: server umount lustre-OST0000 complete [10422.243167] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10422.255228] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10422.270940] Lustre: Skipped 1 previous similar message [10427.854909] Lustre: server umount lustre-MDT0000 complete [10431.480960] LustreError: 379806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250854 with bad export cookie 1224588565813979332 [10431.485297] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10431.497645] LustreError: 379806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10437.923719] Lustre: server umount lustre-MDT0001 complete [10447.420518] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10450.035987] Key type lgssc unregistered [10450.301371] LNet: 384649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10450.307617] LNetError: 384649:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10450.334839] LNet: Removed LNI 192.168.203.145@tcp [10451.151662] Key type .llcrypt unregistered [10451.154752] Key type ._llcrypt unregistered [10496.559978] Key type ._llcrypt registered [10496.562907] Key type .llcrypt registered [10496.679658] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10497.743892] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10497.780801] alg: No test for adler32 (adler32-zlib) [10498.762684] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [10499.038933] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10500.847181] Key type lgssc registered [10502.208172] Lustre: Echo OBD driver; http://www.lustre.org/ [10512.329701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10512.366153] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10513.675026] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10513.719389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10513.943158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10519.427897] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10528.089101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10528.158707] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10528.184553] Lustre: 387447:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10528.621273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10532.677723] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10538.899477] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10544.987631] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10549.216154] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10549.233980] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10549.264427] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10551.780776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10551.782535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.530874] Lustre: server umount lustre-MDT0000 complete [10557.502459] LustreError: 387439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782250980 with bad export cookie 18439611369564224707 [10557.523718] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10557.528022] LustreError: 387439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10557.921899] Lustre: server umount lustre-MDT0001 complete [10567.705707] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 17:43:08 (1782250988) [10574.237501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10574.672200] LustreError: 389958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10574.687192] LustreError: 389958:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10574.738718] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10578.673818] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10579.937305] LustreError: 389959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10585.061570] LustreError: 389958:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10586.026160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10586.377700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10590.549428] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10598.956550] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10605.028948] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10611.859501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10612.103224] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10612.371368] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10618.780590] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10625.991807] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10634.137794] Lustre: Permanently deactivating lustre-OST0000 [10634.147408] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10641.492951] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10643.207392] LustreError: 392946:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10643.215383] LustreError: 392946:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [10643.222980] LustreError: 392946:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10646.830102] LustreError: 392994:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [10646.839147] LustreError: 392994:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10655.161571] Lustre: server umount lustre-OST0000 complete [10657.600283] LustreError: 393244:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [10657.610145] LustreError: 393244:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10660.834045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10660.855082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10660.865398] Lustre: Skipped 2 previous similar messages [10665.956306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10665.962559] Lustre: Skipped 1 previous similar message [10670.135378] Lustre: server umount lustre-MDT0000 complete [10671.073707] LustreError: 389959:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10671.109258] LustreError: 389959:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10673.878613] LustreError: 389940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251096 with bad export cookie 18439611369564225288 [10673.894091] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10683.509932] Lustre: server umount lustre-MDT0001 complete [10690.915464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10694.565710] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10697.534836] LustreError: 394402:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10705.671491] Lustre: 394743:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.145@tcp [10708.245935] Lustre: server umount MGS complete [10723.347980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10723.762069] LustreError: 395467:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10723.793218] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10723.803655] Lustre: Skipped 1 previous similar message [10723.868609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10728.536724] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10734.052586] LustreError: 395468:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10734.082716] LustreError: 395468:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10737.564599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10737.881829] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10742.795487] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10749.989780] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10755.440367] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10757.319544] Lustre: Permanently reactivating lustre-OST0000 [10757.342319] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10757.359887] Lustre: Skipped 2 previous similar messages [10786.275239] LustreError: 397275:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10793.187844] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10793.490893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10793.499511] Lustre: Skipped 1 previous similar message [10799.736722] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10807.163599] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10815.969179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10815.983376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10816.002521] Lustre: Skipped 1 previous similar message [10816.008629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10816.992504] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10818.368250] Lustre: server umount lustre-OST0000 complete [10825.184224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10825.196231] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10825.217162] Lustre: Skipped 1 previous similar message [10825.238954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.245195] Lustre: Skipped 2 previous similar messages [10828.772042] Lustre: server umount lustre-MDT0000 complete [10832.355450] LustreError: 395464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10832.383792] LustreError: 395464:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10832.628079] LustreError: 395449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251255 with bad export cookie 18439611369564226338 [10832.640756] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10832.647244] LustreError: 395449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10832.880470] Lustre: server umount lustre-MDT0001 complete [10843.618786] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [10846.552869] Key type lgssc unregistered [10846.867145] LNet: 399879:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10846.874791] LNetError: 399879:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10846.903687] LNet: Removed LNI 192.168.203.145@tcp [10847.978283] Key type .llcrypt unregistered [10847.981253] Key type ._llcrypt unregistered [10865.418184] Key type ._llcrypt registered [10865.419970] Key type .llcrypt registered [10865.524993] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [10879.767351] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [10881.126420] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10881.269395] alg: No test for adler32 (adler32-zlib) [10882.411563] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [10882.696089] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [10884.407164] Key type lgssc registered [10885.310321] Lustre: Echo OBD driver; http://www.lustre.org/ [10919.345340] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10919.382687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10920.633379] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10920.662782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10920.759721] Lustre: lustre-MDT0000: new disk, initializing [10920.843636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10920.859661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10924.891579] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10935.322521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10935.458248] Lustre: 403037:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10935.524497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10935.527273] Lustre: Skipped 1 previous similar message [10935.582722] Lustre: lustre-MDT0001: new disk, initializing [10935.662963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10935.695252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10935.712650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10940.030558] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10949.159327] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10954.729577] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10962.392057] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10962.732809] Lustre: lustre-OST0000: new disk, initializing [10962.739526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10962.752079] Lustre: 405002:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10962.931820] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10970.171551] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [10972.253209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10972.262435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10972.398370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10981.058378] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10986.528383] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10986.850751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10993.035316] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10993.367634] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10996.198292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10996.216224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10996.219169] Lustre: Skipped 1 previous similar message [10996.241762] Lustre: Skipped 1 previous similar message [11001.318183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11001.331542] Lustre: Skipped 1 previous similar message [11001.609936] Lustre: server umount lustre-OST0000 complete [11006.433753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11006.437706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11006.444160] Lustre: Skipped 1 previous similar message [11006.456714] Lustre: Skipped 1 previous similar message [11011.641835] Lustre: server umount lustre-MDT0000 complete [11015.329938] LustreError: 403028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251438 with bad export cookie 15514848361154250967 [11015.330431] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11015.352098] LustreError: 403028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11015.667761] Lustre: server umount lustre-MDT0001 complete [11024.482634] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11026.036248] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 17:50:46 (1782251446) [11032.162583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11032.546986] LustreError: 407463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11032.567323] LustreError: 407463:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11032.634061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11036.986697] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11037.680263] LustreError: 407464:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11042.786863] LustreError: 407463:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11045.129037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11045.511218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11050.364190] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11057.226804] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11062.408174] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11068.486229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11068.749367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11074.263903] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11082.295883] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11085.000014] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11087.588611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11096.554368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11096.558526] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11096.577742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11098.594108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11098.600314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11099.690547] Lustre: server umount lustre-OST0000 complete [11103.712700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11103.717184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11103.728604] Lustre: Skipped 1 previous similar message [11103.751784] Lustre: Skipped 3 previous similar messages [11109.715230] Lustre: server umount lustre-MDT0000 complete [11113.439297] LustreError: 407443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251536 with bad export cookie 15514848361154251849 [11113.459436] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11113.762913] Lustre: server umount lustre-MDT0001 complete [11124.688474] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11127.700490] Key type lgssc unregistered [11127.984925] LNet: 411192:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11127.996419] LNetError: 411192:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11129.060909] LNet: Removed LNI 192.168.203.145@tcp [11130.046785] Key type .llcrypt unregistered [11130.049533] Key type ._llcrypt unregistered [11154.013236] Key type ._llcrypt registered [11154.016586] Key type .llcrypt registered [11154.160434] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11154.454417] Key type .llcrypt unregistered [11154.457958] Key type ._llcrypt unregistered [11172.430839] Key type ._llcrypt registered [11172.433674] Key type .llcrypt registered [11172.553596] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11173.797585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11173.834435] alg: No test for adler32 (adler32-zlib) [11174.912537] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [11175.207877] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [11176.871208] Key type lgssc registered [11178.227035] Lustre: Echo OBD driver; http://www.lustre.org/ [11185.308996] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11185.347045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11186.784889] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11190.672844] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11196.385707] LustreError: 413403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11196.401992] LustreError: 413403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11197.392942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11197.646049] LustreError: 413398:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11197.668624] LustreError: 413398:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11197.681867] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11202.886492] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11203.043102] LustreError: 413404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.064573] LustreError: 413404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11208.164579] LustreError: 413403:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11208.187892] LustreError: 413403:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11209.172487] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11212.261167] LustreError: 413404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11212.300288] LustreError: 413404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11212.301714] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11215.470941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11215.766818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11219.368501] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11227.619328] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11227.627280] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11227.643981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11228.573163] Lustre: server umount lustre-OST0000 complete [11234.789882] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11234.800821] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11234.816307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11236.322579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11236.323852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11238.160429] Lustre: server umount lustre-MDT0000 complete [11241.444747] LustreError: 413404:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11241.455428] LustreError: 413404:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11241.541042] LustreError: 413385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251664 with bad export cookie 4081499985755800793 [11241.541826] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11241.553692] LustreError: 413385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11252.472707] Lustre: server umount lustre-MDT0001 complete [11264.437326] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11267.619374] Key type lgssc unregistered [11267.943816] LNet: 416552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11267.954730] LNetError: 416552:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11267.975676] LNet: Removed LNI 192.168.203.145@tcp [11269.069242] Key type .llcrypt unregistered [11269.073491] Key type ._llcrypt unregistered [11293.568207] Key type ._llcrypt registered [11293.570300] Key type .llcrypt registered [11293.724944] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11295.219450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11295.278102] alg: No test for adler32 (adler32-zlib) [11296.516974] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [11296.911172] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [11298.616928] Key type lgssc registered [11300.108456] Lustre: Echo OBD driver; http://www.lustre.org/ [11308.538823] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11308.588704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11324.191409] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11329.311521] LustreError: 417944:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.145@tcp: failed processing log, type 4: rc = -110 [11355.999759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11362.772341] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11370.233599] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11376.933954] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.254082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11377.343661] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11382.308657] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11382.756185] LustreError: 417969:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11382.779767] LustreError: 417969:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11387.874322] LustreError: 417970:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.110596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11390.382774] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11394.726338] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11402.107930] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11409.758775] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11418.083774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11418.091332] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11418.104329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11421.155286] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11421.159896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11421.172047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11422.180087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11422.188351] Lustre: Skipped 1 previous similar message [11424.230858] Lustre: server umount lustre-OST0000 complete [11431.392497] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11431.401481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11431.424892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11433.999852] Lustre: server umount lustre-MDT0000 complete [11437.425745] LustreError: 417950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251860 with bad export cookie 1600249982538883637 [11437.432971] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11437.441699] LustreError: 417950:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11437.692768] Lustre: server umount lustre-MDT0001 complete [11451.210527] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11455.406954] Key type lgssc unregistered [11455.758914] LNet: 421519:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11455.770462] LNetError: 421519:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11455.806946] LNet: Removed LNI 192.168.203.145@tcp [11456.973850] Key type .llcrypt unregistered [11456.977482] Key type ._llcrypt unregistered [11481.627643] Key type ._llcrypt registered [11481.629619] Key type .llcrypt registered [11481.793991] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11482.990852] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11483.081830] alg: No test for adler32 (adler32-zlib) [11484.110307] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [11484.305773] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [11486.015218] Key type lgssc registered [11487.325476] Lustre: Echo OBD driver; http://www.lustre.org/ [11494.539777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11494.568682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11495.976903] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11500.099723] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11506.146861] LustreError: 422938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11506.172645] LustreError: 422938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11508.132124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11508.617638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11513.356924] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11519.709282] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11524.901291] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11531.247533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11531.575111] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11537.271601] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11540.793115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11543.715645] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11551.206176] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11551.213637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11551.239855] Lustre: Skipped 2 previous similar messages [11551.257959] Lustre: Skipped 2 previous similar messages [11551.725191] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.45@tcp (stopping) [11552.060866] Lustre: server umount lustre-MDT0000 complete [11555.013770] LustreError: 422919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.45@tcp arrived at 1782251977 with bad export cookie 18120910989632142677 [11555.039654] LustreError: 422919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11556.332388] LustreError: 422938:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11556.366208] LustreError: 422938:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11557.041775] LustreError: 422920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782251979 with bad export cookie 18120910989632141781 [11557.046550] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11557.056879] LustreError: 422920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11559.392984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11559.413446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11559.442982] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11561.443828] LustreError: 422939:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11561.455536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11561.485814] LustreError: 422939:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11563.323750] Lustre: server umount lustre-OST0000 complete [11566.561027] LustreError: 425323:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11566.570754] LustreError: 425323:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [11569.312214] Lustre: server umount lustre-MDT0001 complete [11579.455454] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11582.703321] Key type lgssc unregistered [11583.166725] LNet: 426621:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11583.181793] LNetError: 426621:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11583.214678] LNet: Removed LNI 192.168.203.145@tcp [11584.295919] Key type .llcrypt unregistered [11584.298082] Key type ._llcrypt unregistered [11610.886718] Key type ._llcrypt registered [11610.894086] Key type .llcrypt registered [11611.097758] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11612.436194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11612.512295] alg: No test for adler32 (adler32-zlib) [11613.748407] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [11613.952532] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [11615.671526] Key type lgssc registered [11616.962935] Lustre: Echo OBD driver; http://www.lustre.org/ [11624.740650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11624.751305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.216089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11631.101100] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11635.680785] LustreError: 428042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.712607] LustreError: 428042:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11639.896213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11640.189134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11645.006031] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11652.592324] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11658.672247] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11664.851099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11665.189728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11670.344795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11671.379584] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11678.163595] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11682.617875] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11682.646487] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11682.664302] Lustre: Skipped 1 previous similar message [11685.838810] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [11687.907552] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11687.918876] Lustre: Skipped 1 previous similar message [11689.012904] Lustre: server umount lustre-MDT0001 complete [11693.024512] LustreError: 428036:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11693.057453] LustreError: 428036:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11696.096970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11696.106353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11696.119555] Lustre: Skipped 1 previous similar message [11696.134138] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11698.146747] LustreError: 428042:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11701.694072] Lustre: server umount lustre-OST0000 complete [11703.274382] LustreError: 428041:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11705.873663] Lustre: server umount lustre-MDT0000 complete [11719.168813] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11721.733683] Key type lgssc unregistered [11721.986198] LNet: 431739:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11722.002917] LNetError: 431739:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11722.019812] LNet: Removed LNI 192.168.203.145@tcp [11723.240112] Key type .llcrypt unregistered [11723.243766] Key type ._llcrypt unregistered [11750.188692] Key type ._llcrypt registered [11750.190041] Key type .llcrypt registered [11750.332601] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [11766.767832] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11768.049515] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11768.244152] alg: No test for adler32 (adler32-zlib) [11769.412841] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [11769.835762] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [11771.671176] Key type lgssc registered [11772.828626] Lustre: Echo OBD driver; http://www.lustre.org/ [11813.267617] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [11830.270289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11830.320547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11831.811166] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11831.933195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11832.034954] Lustre: lustre-MDT0000: new disk, initializing [11832.126319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11832.145945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11838.408347] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11855.800163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11856.199844] Lustre: lustre-OST0000: new disk, initializing [11856.203083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11856.208764] Lustre: Skipped 1 previous similar message [11856.216264] Lustre: 436778:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11856.344434] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11862.711253] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11864.222232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11864.230834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11864.278491] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11871.663696] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11885.877344] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11886.006857] Lustre: 435761:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11886.112972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11886.178444] Lustre: lustre-MDT0001: new disk, initializing [11886.294161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11886.334312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11886.342321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11890.994371] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [11900.268681] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11900.583210] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11905.515587] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11905.721762] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11911.650989] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11911.676908] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11911.686478] Lustre: Skipped 1 previous similar message [11913.197718] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.45@tcp (stopping) [11916.223921] Lustre: server umount lustre-MDT0001 complete [11921.900123] LustreError: 435769:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11921.926846] LustreError: 435769:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [11927.014383] LustreError: 435767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11927.061969] LustreError: 435767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11932.133459] LustreError: 435767:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11932.156543] LustreError: 435767:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11936.226650] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11936.235582] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11936.262148] Lustre: Skipped 1 previous similar message [11936.272117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11937.263152] LustreError: 435773:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11940.536681] Lustre: server umount lustre-OST0000 complete [11942.371931] LustreError: 435772:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11950.809770] Lustre: server umount lustre-MDT0000 complete [11966.676919] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [11969.761729] Key type lgssc unregistered [11970.112442] LNet: 440525:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11970.122699] LNetError: 440525:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11970.144549] LNet: Removed LNI 192.168.203.145@tcp [11971.210373] Key type .llcrypt unregistered [11971.215070] Key type ._llcrypt unregistered [12012.646457] Key type ._llcrypt registered [12012.648682] Key type .llcrypt registered [12012.758857] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [12012.989290] Key type .llcrypt unregistered [12012.993460] Key type ._llcrypt unregistered [12028.682354] Key type ._llcrypt registered [12028.685576] Key type .llcrypt registered [12028.871886] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [12029.667555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12029.685189] alg: No test for adler32 (adler32-zlib) [12030.634135] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [12030.797077] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [12032.479226] Key type lgssc registered [12033.856550] Lustre: Echo OBD driver; http://www.lustre.org/ [12064.551309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12064.577277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12065.926197] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12065.955254] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12066.064208] Lustre: lustre-MDT0000: new disk, initializing [12066.204626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12066.224200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12070.935328] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12080.962573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12081.121204] Lustre: 443899:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12081.217339] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [12081.222394] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12081.237116] Lustre: Skipped 1 previous similar message [12081.302915] Lustre: lustre-MDT0001: new disk, initializing [12081.408642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12081.456508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12081.478951] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12086.738172] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12096.999737] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12103.302373] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12108.924190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12109.203403] Lustre: lustre-OST0000: new disk, initializing [12109.208541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12109.227811] Lustre: 445873:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12109.362621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12113.863914] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12118.161753] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12118.165125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12118.269534] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12122.762652] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12128.745258] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12128.749249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12128.765692] Lustre: Skipped 1 previous similar message [12128.776546] Lustre: Skipped 2 previous similar messages [12133.867393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12133.878741] Lustre: Skipped 2 previous similar messages [12134.984573] Lustre: server umount lustre-MDT0000 complete [12138.552117] LustreError: 443892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252561 with bad export cookie 14814666053866365099 [12138.563062] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12138.804369] Lustre: server umount lustre-MDT0001 complete [12152.299556] Lustre: server umount lustre-OST0000 complete [12162.467834] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 18:09:43 (1782252583) [12168.831298] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12169.300180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12174.307149] LustreError: 447900:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12174.343250] LustreError: 447900:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12174.415868] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12178.401846] LustreError: 447901:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12182.124228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12182.476252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12187.130324] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12194.108328] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12198.809695] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12217.254750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12217.485985] Lustre: Found index 0 for lustre-OST0000, updating log [12217.706948] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12223.380245] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12229.920629] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12240.253852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12249.576531] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12249.583131] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12249.605935] Lustre: Skipped 1 previous similar message [12249.614043] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12251.836721] Lustre: server umount lustre-OST0000 complete [12266.976307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12266.994269] Lustre: Skipped 1 previous similar message [12266.998822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12267.012294] Lustre: Skipped 4 previous similar messages [12277.224478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12277.237207] Lustre: Skipped 3 previous similar messages [12280.675900] LNet: 1 local NIs in recovery (showing 1): 192.168.203.145@tcp [12281.311380] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12281.545754] Lustre: server umount lustre-MDT0000 complete [12283.362597] LustreError: 448629:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12283.378564] LustreError: 448629:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12285.229844] LustreError: 447880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782252707 with bad export cookie 14814666053866366492 [12285.232747] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12285.247968] LustreError: 447880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12285.643723] Lustre: server umount lustre-MDT0001 complete [12294.088380] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 18:11:54 (1782252714) [12912.475230] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12912.876082] LustreError: 454340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12912.971226] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12916.762505] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12918.241966] LustreError: 454341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12922.337692] LustreError: 454340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12923.691522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12924.021235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12927.784467] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12934.928397] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12940.576969] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12960.996308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12961.195103] Lustre: Found index 0 for lustre-OST0000, updating log [12961.262129] LustreError: 456167:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12961.276198] LustreError: 456167:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12961.434805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12967.020712] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [12967.936754] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [12972.895703] LNet: 1 local NIs in recovery (showing 1): 192.168.203.145@tcp [12973.451485] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12976.291511] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12976.302492] Lustre: Skipped 1 previous similar message [12976.306973] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12976.314605] Lustre: Skipped 2 previous similar messages [12981.503843] Lustre: server umount lustre-OST0000 complete [12985.312528] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12985.322055] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12985.352791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12986.403228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12989.477982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12989.486965] Lustre: Skipped 7 previous similar messages [12991.522302] LustreError: 454340:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12991.536577] LustreError: 454340:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12991.552096] Lustre: server umount lustre-MDT0000 complete [12994.874110] LustreError: 454321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253417 with bad export cookie 14814666053866367577 [12994.880787] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12994.887077] LustreError: 454321:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12995.235613] Lustre: server umount lustre-MDT0001 complete [13023.208665] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [13032.584164] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [13064.007635] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 18:24:44 (1782253484) [13081.878364] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [13084.731493] Key type lgssc unregistered [13085.069783] LNet: 460949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13085.088146] LNetError: 460949:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13215.163466] LNet: Removed LNI 192.168.203.145@tcp [13216.058785] Key type .llcrypt unregistered [13216.063569] Key type ._llcrypt unregistered [13233.294668] Key type ._llcrypt registered [13233.298153] Key type .llcrypt registered [13233.418043] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [13234.603827] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13234.622949] alg: No test for adler32 (adler32-zlib) [13235.689805] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [13235.911830] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [13237.639170] Key type lgssc registered [13238.803780] Lustre: Echo OBD driver; http://www.lustre.org/ [13243.176805] LNet: Added LNI 192.168.203.145@tcp100 [8/256/0/180] [13244.171567] LNet: Added LNI 192.168.203.145@tcp101 [8/256/0/180] [13245.166596] LNet: Added LNI 192.168.203.145@tcp102 [8/256/0/180] [13246.198645] LNet: Added LNI 192.168.203.145@tcp103 [8/256/0/180] [13247.062869] LNet: Added LNI 192.168.203.145@tcp104 [8/256/0/180] [13247.909311] LNet: Added LNI 192.168.203.145@tcp105 [8/256/0/180] [13248.784853] LNet: Added LNI 192.168.203.145@tcp106 [8/256/0/180] [13249.779832] LNet: Added LNI 192.168.203.145@tcp107 [8/256/0/180] [13250.776459] LNet: Added LNI 192.168.203.145@tcp108 [8/256/0/180] [13251.661292] LNet: Added LNI 192.168.203.145@tcp109 [8/256/0/180] [13252.809119] LNet: Added LNI 192.168.203.145@tcp110 [8/256/0/180] [13253.814160] LNet: Added LNI 192.168.203.145@tcp111 [8/256/0/180] [13254.623130] LNet: Added LNI 192.168.203.145@tcp112 [8/256/0/180] [13255.521134] LNet: Added LNI 192.168.203.145@tcp113 [8/256/0/180] [13256.585527] LNet: Added LNI 192.168.203.145@tcp114 [8/256/0/180] [13257.478606] LNet: Added LNI 192.168.203.145@tcp115 [8/256/0/180] [13258.357193] LNet: Added LNI 192.168.203.145@tcp116 [8/256/0/180] [13259.141301] LNet: Added LNI 192.168.203.145@tcp117 [8/256/0/180] [13260.077509] LNet: Added LNI 192.168.203.145@tcp118 [8/256/0/180] [13260.996535] LNet: Added LNI 192.168.203.145@tcp119 [8/256/0/180] [13261.763501] LNet: Added LNI 192.168.203.145@tcp120 [8/256/0/180] [13262.708550] LNet: Added LNI 192.168.203.145@tcp121 [8/256/0/180] [13263.793608] LNet: Added LNI 192.168.203.145@tcp122 [8/256/0/180] [13264.769664] LNet: Added LNI 192.168.203.145@tcp123 [8/256/0/180] [13265.609562] LNet: Added LNI 192.168.203.145@tcp124 [8/256/0/180] [13266.549431] LNet: Added LNI 192.168.203.145@tcp125 [8/256/0/180] [13267.496140] LNet: Added LNI 192.168.203.145@tcp126 [8/256/0/180] [13268.444785] LNet: Added LNI 192.168.203.145@tcp127 [8/256/0/180] [13269.284501] LNet: Added LNI 192.168.203.145@tcp128 [8/256/0/180] [13270.180876] LNet: Added LNI 192.168.203.145@tcp129 [8/256/0/180] [13271.088775] LNet: Added LNI 192.168.203.145@tcp130 [8/256/0/180] [13271.895697] LNet: Added LNI 192.168.203.145@tcp131 [8/256/0/180] [13272.769296] LNet: Added LNI 192.168.203.145@tcp132 [8/256/0/180] [13273.455217] LNet: Added LNI 192.168.203.145@tcp133 [8/256/0/180] [13274.241393] LNet: Added LNI 192.168.203.145@tcp134 [8/256/0/180] [13275.104424] LNet: Added LNI 192.168.203.145@tcp135 [8/256/0/180] [13281.462875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13281.492643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13282.639143] Lustre: 463982:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [13282.732183] LNetError: 461442:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [13282.739416] LNetError: Protocol error connecting to 192.168.203.145@tcp118 at host 192.168.203.145:988: is it running a compatible version of Lustre? [13282.748695] LNetError: Skipped 6 previous similar messages [13282.834767] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13282.860298] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13282.968291] Lustre: lustre-MDT0000: new disk, initializing [13283.040400] LNet: 8 local NIs in recovery (showing 8): 192.168.203.145@tcp120, 192.168.203.145@tcp121, 192.168.203.145@tcp122, 192.168.203.145@tcp124, 192.168.203.145@tcp126, 192.168.203.145@tcp125, 192.168.203.145@tcp129, 192.168.203.145@tcp128 [13283.082613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13283.101748] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13285.092884] LNetError: 461440:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [13285.116735] LNetError: 461440:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 20 previous similar messages [13285.130252] LNetError: Protocol error connecting to 192.168.203.145@tcp121 at host 192.168.203.145:988: is it running a compatible version of Lustre? [13285.148214] LNetError: Skipped 14 previous similar messages [13287.813936] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13289.191846] LNetError: 464009:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [13289.211377] LNetError: 464009:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [13289.235220] LNetError: Protocol error connecting to 192.168.203.145@tcp121 at host 192.168.203.145:988: is it running a compatible version of Lustre? [13297.383872] LNetError: 464004:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [13297.398278] LNetError: Protocol error connecting to 192.168.203.145@tcp121 at host 192.168.203.145:988: is it running a compatible version of Lustre? [13298.140559] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13298.320507] Lustre: 463999:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13298.401892] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13298.415296] Lustre: Skipped 1 previous similar message [13298.517581] Lustre: lustre-MDT0001: new disk, initializing [13298.564976] Lustre: 464800:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [13298.578986] Lustre: 464800:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13298.708756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13298.753130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13298.765329] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13303.210933] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13312.767912] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13313.763852] LNetError: 461442:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [13313.774810] LNetError: Protocol error connecting to 192.168.203.145@tcp121 at host 192.168.203.145:988: is it running a compatible version of Lustre? [13317.785789] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13321.954376] LNetError: 461439:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [13321.973854] LNetError: 461439:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 5 previous similar messages [13321.982172] LNetError: Protocol error connecting to 192.168.203.145@tcp124 at host 192.168.203.145:988: is it running a compatible version of Lustre? [13321.995633] LNetError: Skipped 5 previous similar messages [13323.638769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13323.953431] Lustre: lustre-OST0000: new disk, initializing [13323.965604] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13323.983637] Lustre: 465973:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13324.157253] Lustre: 465978:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [13324.170621] Lustre: 465978:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [13324.237693] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13330.476650] Lustre: 466322:0:(lustre_peer.c:105:class_add_uuid()) 192.168.203.145@tcp: can't add NID 192.168.203.145@tcp130: rc = -75 [13330.486288] Lustre: 466322:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [13330.493646] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13330.542259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13330.554481] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13330.679969] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13339.327375] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13339.366996] LNetError: 464008:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [13339.378126] LNetError: 464008:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 9 previous similar messages [13339.385706] LNetError: Protocol error connecting to 192.168.203.145@tcp125 at host 192.168.203.145:988: is it running a compatible version of Lustre? [13339.399060] LNetError: Skipped 9 previous similar messages [13342.431190] LNet: 1 local NIs in recovery (showing 1): 192.168.203.145@tcp121 [13346.277318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13346.283791] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13346.292607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13346.296496] Lustre: Skipped 1 previous similar message [13346.318110] Lustre: Skipped 1 previous similar message [13351.394408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13351.404698] Lustre: Skipped 1 previous similar message [13352.137234] Lustre: server umount lustre-OST0000 complete [13356.513701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13356.530200] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13356.536103] Lustre: Skipped 1 previous similar message [13361.632525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13361.639619] Lustre: Skipped 1 previous similar message [13362.075788] Lustre: server umount lustre-MDT0000 complete [13365.796283] LustreError: 463991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782253788 with bad export cookie 12436123895186725452 [13365.804254] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13365.808479] LustreError: 463991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13366.168196] Lustre: server umount lustre-MDT0001 complete [13376.612134] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [13378.285951] LNetError: 464012:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.145 [13378.301423] LNetError: 464012:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 17 previous similar messages [13378.328647] LNetError: Protocol error connecting to 192.168.203.145@tcp120 at host 192.168.203.145:988: is it running a compatible version of Lustre? [13378.342623] LNetError: Skipped 17 previous similar messages [13379.240363] Key type lgssc unregistered [13379.658624] LNet: 467889:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13379.680950] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13379.692341] LNet: Removed LNI 192.168.203.145@tcp [13380.767319] LNet: Removed LNI 192.168.203.145@tcp100 [13380.769429] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13380.789938] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13380.799643] LNet: Removed LNI 192.168.203.145@tcp101 [13381.855714] LNet: Removed LNI 192.168.203.145@tcp102 [13381.861150] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13381.873806] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13381.884923] LNet: Removed LNI 192.168.203.145@tcp103 [13382.943414] LNet: Removed LNI 192.168.203.145@tcp104 [13383.967239] LNet: Removed LNI 192.168.203.145@tcp105 [13383.969673] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13383.977159] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13383.990363] LNet: Removed LNI 192.168.203.145@tcp106 [13385.060771] LNet: Removed LNI 192.168.203.145@tcp107 [13386.079239] LNet: Removed LNI 192.168.203.145@tcp108 [13387.106046] LNet: Removed LNI 192.168.203.145@tcp109 [13388.127183] LNet: Removed LNI 192.168.203.145@tcp110 [13388.131192] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13388.138592] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13388.146604] LNet: Removed LNI 192.168.203.145@tcp111 [13389.215145] LNet: Removed LNI 192.168.203.145@tcp112 [13390.239164] LNet: Removed LNI 192.168.203.145@tcp113 [13391.263367] LNet: Removed LNI 192.168.203.145@tcp114 [13392.287208] LNet: Removed LNI 192.168.203.145@tcp115 [13393.311188] LNet: Removed LNI 192.168.203.145@tcp116 [13394.335286] LNet: Removed LNI 192.168.203.145@tcp117 [13395.359139] LNet: Removed LNI 192.168.203.145@tcp118 [13396.383183] LNet: Removed LNI 192.168.203.145@tcp119 [13396.386110] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13396.394761] LNetError: 467889:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13396.412633] LNet: Removed LNI 192.168.203.145@tcp120 [13396.417564] LNet: Removed LNI 192.168.203.145@tcp121 [13397.471240] LNet: Removed LNI 192.168.203.145@tcp122 [13398.497403] LNet: Removed LNI 192.168.203.145@tcp123 [13399.519177] LNet: Removed LNI 192.168.203.145@tcp124 [13400.543136] LNet: Removed LNI 192.168.203.145@tcp125 [13401.567322] LNet: Removed LNI 192.168.203.145@tcp126 [13402.592792] LNet: Removed LNI 192.168.203.145@tcp127 [13403.617399] LNet: Removed LNI 192.168.203.145@tcp128 [13404.639424] LNet: Removed LNI 192.168.203.145@tcp129 [13404.643518] LNet: Removed LNI 192.168.203.145@tcp130 [13404.645973] LNet: Removed LNI 192.168.203.145@tcp131 [13404.650341] LNet: Removed LNI 192.168.203.145@tcp132 [13404.658656] LNet: Removed LNI 192.168.203.145@tcp133 [13404.666149] LNet: Removed LNI 192.168.203.145@tcp134 [13404.678179] LNet: Removed LNI 192.168.203.145@tcp135 [13405.502993] Key type .llcrypt unregistered [13405.509553] Key type ._llcrypt unregistered [13426.346587] Key type ._llcrypt registered [13426.349198] Key type .llcrypt registered [13426.443808] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [13441.594893] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [13442.785920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13442.820282] alg: No test for adler32 (adler32-zlib) [13443.891575] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [13444.175110] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [13445.879640] Key type lgssc registered [13447.040760] Lustre: Echo OBD driver; http://www.lustre.org/ [13485.327590] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 18:31:45 (1782253905) [13503.921587] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [13506.992817] Key type lgssc unregistered [13507.263294] LNet: 471995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13507.279663] LNetError: 471995:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13507.310954] LNet: Removed LNI 192.168.203.145@tcp [13508.691742] Key type .llcrypt unregistered [13508.697410] Key type ._llcrypt unregistered [13525.966370] Key type ._llcrypt registered [13525.968770] Key type .llcrypt registered [13526.105136] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [13527.041417] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13527.145104] alg: No test for adler32 (adler32-zlib) [13528.236816] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [13528.472679] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [13530.169116] Key type lgssc registered [13531.217688] Lustre: Echo OBD driver; http://www.lustre.org/ [13539.471741] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13539.511892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13540.788961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13540.842353] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13540.960682] Lustre: lustre-MDT0000: new disk, initializing [13541.067797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13541.099302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13545.933357] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13555.642039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13555.739734] Lustre: 473209:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13555.791569] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13555.794024] Lustre: Skipped 1 previous similar message [13555.887285] Lustre: lustre-MDT0001: new disk, initializing [13555.990692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13556.020793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13556.032885] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13560.514240] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13570.030816] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13575.599181] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13581.933851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13582.202858] Lustre: lustre-OST0000: new disk, initializing [13582.206794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13582.214409] Lustre: 475177:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13582.297444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13587.778270] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13590.115097] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13590.125622] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13590.252877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13598.083702] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [13602.018674] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing lnet_if_list [13606.669562] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13606.696387] LNet: Added LNI 192.168.203.145@tcp10 [8/256/0/180] [13612.368949] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13612.399937] LNet: Added LNI 192.168.203.145@tcp11 [8/256/0/180] [13616.955887] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13616.989601] LNet: Added LNI 192.168.203.145@tcp12 [8/256/0/180] [13621.258488] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13621.307412] LNet: Added LNI 192.168.203.145@tcp13 [8/256/0/180] [13626.224590] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13626.263208] LNet: Added LNI 192.168.203.145@tcp14 [8/256/0/180] [13630.800642] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13630.844194] LNet: Added LNI 192.168.203.145@tcp15 [8/256/0/180] [13636.027678] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13636.084307] LNet: Added LNI 192.168.203.145@tcp16 [8/256/0/180] [13640.875458] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13640.900747] LNet: Added LNI 192.168.203.145@tcp17 [8/256/0/180] [13645.691315] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13645.739650] LNet: Added LNI 192.168.203.145@tcp18 [8/256/0/180] [13651.292818] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13651.324467] LNet: Added LNI 192.168.203.145@tcp19 [8/256/0/180] [13657.195742] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13657.236535] LNet: Added LNI 192.168.203.145@tcp20 [8/256/0/180] [13663.288229] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13663.334325] LNet: Added LNI 192.168.203.145@tcp21 [8/256/0/180] [13668.310526] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13668.327305] LNet: Added LNI 192.168.203.145@tcp22 [8/256/0/180] [13672.711800] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13672.733191] LNet: Added LNI 192.168.203.145@tcp23 [8/256/0/180] [13677.103816] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13677.147269] LNet: Added LNI 192.168.203.145@tcp24 [8/256/0/180] [13681.961546] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13681.993933] LNet: Added LNI 192.168.203.145@tcp25 [8/256/0/180] [13686.492923] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13686.529188] LNet: Added LNI 192.168.203.145@tcp26 [8/256/0/180] [13692.882151] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13692.927319] LNet: Added LNI 192.168.203.145@tcp27 [8/256/0/180] [13698.385513] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13698.426823] LNet: Added LNI 192.168.203.145@tcp28 [8/256/0/180] [13703.909923] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13703.957745] LNet: Added LNI 192.168.203.145@tcp29 [8/256/0/180] [13709.651721] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13709.685485] LNet: Added LNI 192.168.203.145@tcp30 [8/256/0/180] [13715.158845] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13715.190103] LNet: Added LNI 192.168.203.145@tcp31 [8/256/0/180] [13721.043310] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13721.079224] LNet: Added LNI 192.168.203.145@tcp32 [8/256/0/180] [13726.890345] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13726.918936] LNet: Added LNI 192.168.203.145@tcp33 [8/256/0/180] [13733.007332] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13733.042906] LNet: Added LNI 192.168.203.145@tcp34 [8/256/0/180] [13739.415364] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13739.442780] LNet: Added LNI 192.168.203.145@tcp35 [8/256/0/180] [13745.111406] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13745.134443] LNet: Added LNI 192.168.203.145@tcp36 [8/256/0/180] [13751.868285] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13751.896022] LNet: Added LNI 192.168.203.145@tcp37 [8/256/0/180] [13759.377986] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13759.437044] LNet: Added LNI 192.168.203.145@tcp38 [8/256/0/180] [13766.419714] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13766.459522] LNet: Added LNI 192.168.203.145@tcp39 [8/256/0/180] [13773.508913] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13773.565082] LNet: Added LNI 192.168.203.145@tcp40 [8/256/0/180] [13780.427707] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13780.465591] LNet: Added LNI 192.168.203.145@tcp41 [8/256/0/180] [13787.785731] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13787.829167] LNet: Added LNI 192.168.203.145@tcp42 [8/256/0/180] [13795.383574] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13795.414749] LNet: Added LNI 192.168.203.145@tcp43 [8/256/0/180] [13802.535629] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13802.588583] LNet: Added LNI 192.168.203.145@tcp44 [8/256/0/180] [13809.723447] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13809.763461] LNet: Added LNI 192.168.203.145@tcp45 [8/256/0/180] [13817.026472] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13817.075688] LNet: Added LNI 192.168.203.145@tcp46 [8/256/0/180] [13825.311383] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13825.367236] LNet: Added LNI 192.168.203.145@tcp47 [8/256/0/180] [13834.727219] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13834.756727] LNet: Added LNI 192.168.203.145@tcp48 [8/256/0/180] [13843.109413] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13843.129516] LNet: Added LNI 192.168.203.145@tcp49 [8/256/0/180] [13852.701734] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13852.735335] LNet: Added LNI 192.168.203.145@tcp50 [8/256/0/180] [13871.012328] Lustre: server umount lustre-MDT0000 complete [13872.099866] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13872.107783] Lustre: Skipped 1 previous similar message [13874.704499] LustreError: 473201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254297 with bad export cookie 7107819924640418927 [13874.732527] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13877.070407] Lustre: server umount lustre-MDT0001 complete [13892.581726] Lustre: 472480:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782254299/real 1782254299] req@000000007acb09de x1868828719080064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782254315 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13892.618316] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13892.635325] Lustre: Skipped 1 previous similar message [13895.650292] Lustre: 472481:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782254302/real 1782254302] req@0000000028828b3a x1868828719080192/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782254318 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13896.118694] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [13898.786488] Lustre: 472481:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782254305/real 1782254305] req@0000000099673203 x1868828719080320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782254321 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13905.762780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13905.887166] Lustre: 472480:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782254312/real 1782254312] req@000000000f992fe3 x1868828719080704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782254328 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13905.924545] Lustre: 472480:0:(client.c:2489:ptlrpc_expire_one_request()) Skipped 1 previous similar message [13919.265966] LustreError: 472478:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007782c738 x1868828719081728/t0(0) o250->MGC192.168.203.145@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13919.564592] LustreError: 475171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13919.596461] LustreError: 475171:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [13919.636316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13923.585122] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13923.683727] LustreError: 475172:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13928.930497] LustreError: 475171:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13931.189770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13931.483638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13935.569358] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [13936.624986] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13937.703553] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13941.932140] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [13947.158654] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [13965.795970] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13965.810448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13965.827681] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13967.328741] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13967.342306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13968.366046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13968.375493] Lustre: Skipped 1 previous similar message [13971.503537] Lustre: server umount lustre-OST0000 complete [13977.569026] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13977.572957] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13977.590910] Lustre: Skipped 1 previous similar message [13977.597649] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13981.024933] Lustre: server umount lustre-MDT0000 complete [13983.712668] LustreError: 499423:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13983.725115] LustreError: 499423:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [13984.286299] LustreError: 473200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254407 with bad export cookie 7107819924640421454 [13984.299169] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13984.305943] LustreError: 473200:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13984.719423] Lustre: server umount lustre-MDT0001 complete [13995.521971] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [13998.371686] Key type lgssc unregistered [13998.772890] LNet: 502210:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13998.789902] LNetError: 502210:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [13998.797360] LNet: Removed LNI 192.168.203.145@tcp [13998.800601] LNet: Removed LNI 192.168.203.145@tcp10 [13998.803873] LNet: Removed LNI 192.168.203.145@tcp11 [13998.808190] LNet: Removed LNI 192.168.203.145@tcp12 [13998.811428] LNet: Removed LNI 192.168.203.145@tcp13 [13998.817058] LNet: Removed LNI 192.168.203.145@tcp14 [13998.820752] LNet: Removed LNI 192.168.203.145@tcp15 [13998.822976] LNet: Removed LNI 192.168.203.145@tcp16 [13998.826602] LNet: Removed LNI 192.168.203.145@tcp17 [13998.830306] LNet: Removed LNI 192.168.203.145@tcp18 [13998.839671] LNet: Removed LNI 192.168.203.145@tcp19 [13998.843222] LNet: Removed LNI 192.168.203.145@tcp20 [13998.857054] LNet: Removed LNI 192.168.203.145@tcp21 [13998.859294] LNet: Removed LNI 192.168.203.145@tcp22 [13998.864706] LNet: Removed LNI 192.168.203.145@tcp23 [13998.868793] LNet: Removed LNI 192.168.203.145@tcp24 [13998.874708] LNet: Removed LNI 192.168.203.145@tcp25 [13998.878370] LNet: Removed LNI 192.168.203.145@tcp26 [13998.882811] LNet: Removed LNI 192.168.203.145@tcp27 [13998.885405] LNet: Removed LNI 192.168.203.145@tcp28 [13998.891569] LNet: Removed LNI 192.168.203.145@tcp29 [13998.901072] LNet: Removed LNI 192.168.203.145@tcp30 [13998.911156] LNet: Removed LNI 192.168.203.145@tcp31 [13998.915627] LNet: Removed LNI 192.168.203.145@tcp32 [13998.923822] LNet: Removed LNI 192.168.203.145@tcp33 [13998.930628] LNet: Removed LNI 192.168.203.145@tcp34 [13998.936849] LNet: Removed LNI 192.168.203.145@tcp35 [13998.938379] LNet: Removed LNI 192.168.203.145@tcp36 [13998.945987] LNet: Removed LNI 192.168.203.145@tcp37 [13998.951953] LNet: Removed LNI 192.168.203.145@tcp38 [13998.954183] LNet: Removed LNI 192.168.203.145@tcp39 [13998.958273] LNet: Removed LNI 192.168.203.145@tcp40 [13998.961785] LNet: Removed LNI 192.168.203.145@tcp41 [13998.965630] LNet: Removed LNI 192.168.203.145@tcp42 [13998.971673] LNet: Removed LNI 192.168.203.145@tcp43 [13998.982798] LNet: Removed LNI 192.168.203.145@tcp44 [13998.986634] LNet: Removed LNI 192.168.203.145@tcp45 [13998.992090] LNet: Removed LNI 192.168.203.145@tcp46 [13998.999647] LNet: Removed LNI 192.168.203.145@tcp47 [13999.006453] LNet: Removed LNI 192.168.203.145@tcp48 [13999.016172] LNet: Removed LNI 192.168.203.145@tcp49 [13999.031698] LNet: Removed LNI 192.168.203.145@tcp50 [13999.846385] Key type .llcrypt unregistered [13999.851911] Key type ._llcrypt unregistered [14019.504912] Key type ._llcrypt registered [14019.509849] Key type .llcrypt registered [14019.648559] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [14036.076589] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14037.627937] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14037.692047] alg: No test for adler32 (adler32-zlib) [14038.948817] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [14039.311240] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14041.079171] Key type lgssc registered [14042.159314] Lustre: Echo OBD driver; http://www.lustre.org/ [14078.017699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14078.055142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14079.223633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14079.241535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14079.322930] Lustre: lustre-MDT0000: new disk, initializing [14079.431758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14079.450306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14083.518155] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14093.798841] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14093.938891] Lustre: 505406:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14093.991601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14093.996430] Lustre: Skipped 1 previous similar message [14094.069949] Lustre: lustre-MDT0001: new disk, initializing [14094.168318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14094.200436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14094.207101] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14097.922952] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14108.166474] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14114.117045] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14121.238420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14121.412100] Lustre: *** cfs_fail_loc=90f, val=0*** [14121.468276] Lustre: lustre-OST0000: new disk, initializing [14121.474952] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14121.480727] Lustre: 507419:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14121.577321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14127.863773] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14128.694432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14128.717363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14128.874946] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14137.779222] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14149.604360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14149.605097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14149.626938] LustreError: Skipped 1 previous similar message [14149.641517] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14149.654965] Lustre: Skipped 1 previous similar message [14154.726031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14154.740135] Lustre: Skipped 1 previous similar message [14159.327129] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14159.491282] Lustre: server umount lustre-OST0000 complete [14164.966523] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14164.994083] Lustre: Skipped 1 previous similar message [14165.000562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14169.347746] Lustre: server umount lustre-MDT0000 complete [14172.824203] LustreError: 506199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254595 with bad export cookie 8342591525295871236 [14172.827947] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14172.837389] LustreError: 506199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14173.060913] Lustre: server umount lustre-MDT0001 complete [14182.649802] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [14185.169948] Key type lgssc unregistered [14185.447482] LNet: 509326:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14185.460019] LNetError: 509326:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14185.491879] LNet: Removed LNI 192.168.203.145@tcp [14186.392336] Key type .llcrypt unregistered [14186.394708] Key type ._llcrypt unregistered [14205.261440] Key type ._llcrypt registered [14205.264149] Key type .llcrypt registered [14205.368977] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14206.650122] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14206.745584] alg: No test for adler32 (adler32-zlib) [14207.860382] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [14208.088842] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14209.823799] Key type lgssc registered [14210.882836] Lustre: Echo OBD driver; http://www.lustre.org/ [14218.119060] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14218.134394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14219.501463] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14223.452772] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14229.987265] LustreError: 510599:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14229.997422] LustreError: 510599:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [14230.858797] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14231.120994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14235.376886] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14242.332711] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14247.876824] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14255.000060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14255.422421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14262.553861] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14272.009752] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14282.528226] loop0: detected capacity change from 0 to 204800000 [14282.841529] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14282.852782] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14282.862349] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14289.549867] loop0: detected capacity change from 0 to 204800000 [14289.841677] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14289.853401] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14289.863409] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14294.443497] loop0: detected capacity change from 0 to 204800000 [14299.808111] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14299.974695] Lustre: fs15246-OST0000: new disk, initializing [14299.978782] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14299.983910] Lustre: 514189:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14300.039942] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14306.543888] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14315.445393] loop1: detected capacity change from 0 to 204800000 [14320.787718] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14320.926851] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14320.982399] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14321.073949] Lustre: fs15246-MDT0000: new disk, initializing [14321.313874] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14321.345067] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14321.460274] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14322.531654] Lustre: 515312:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000e49efcf8 x1868829430954368/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782254761 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14324.673841] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14331.619043] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14331.626198] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14331.645125] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14348.278310] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14348.320096] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14351.235496] Lustre: server umount fs15246-MDT0000 complete [14353.381687] LustreError: 510595:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14358.500908] LustreError: 510594:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14359.312847] Lustre: server umount fs15246-OST0000 complete [14369.249226] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14369.260799] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14369.288777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14372.326596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14372.333393] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14372.351221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14375.111661] Lustre: server umount lustre-OST0000 complete [14378.978542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14378.994572] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14379.002842] Lustre: Skipped 3 previous similar messages [14384.097204] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14384.106400] Lustre: Skipped 1 previous similar message [14384.917566] Lustre: server umount lustre-MDT0000 complete [14388.293383] LustreError: 510580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782254811 with bad export cookie 9617357402834021200 [14388.298965] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14388.311567] LustreError: 510580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14388.644505] Lustre: server umount lustre-MDT0001 complete [14400.188365] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [14402.480096] Key type lgssc unregistered [14402.855427] LNet: 518407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14402.865727] LNetError: 518407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14402.890716] LNet: Removed LNI 192.168.203.145@tcp [14403.748487] Key type .llcrypt unregistered [14403.755331] Key type ._llcrypt unregistered [14421.194316] Key type ._llcrypt registered [14421.202193] Key type .llcrypt registered [14421.280757] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14422.537157] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14422.614088] alg: No test for adler32 (adler32-zlib) [14423.666793] Lustre: Lustre: Build Version: 2.17.54_83_g947bf2b [14423.878859] LNet: Added LNI 192.168.203.145@tcp [8/256/0/180] [14425.535716] Key type lgssc registered [14426.692779] Lustre: Echo OBD driver; http://www.lustre.org/ [14466.319109] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [14475.808957] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14508.494386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14508.553403] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14509.824888] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14509.857480] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14509.982274] Lustre: lustre-MDT0000: new disk, initializing [14510.133956] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14510.170879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14514.743738] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14525.414947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14525.587682] Lustre: 522873:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14525.667318] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14525.671527] Lustre: Skipped 1 previous similar message [14525.780772] Lustre: lustre-MDT0001: new disk, initializing [14525.859323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14525.888620] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14525.903964] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14530.837614] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14540.092457] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [14545.512131] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [14552.012735] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14552.243769] Lustre: lustre-OST0000: new disk, initializing [14552.247944] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14552.256684] Lustre: 524841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14552.339753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14558.686409] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug -1 all [14559.314402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14559.325168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14559.426617] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14568.440514] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [14573.389607] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14573.581760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14578.096273] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14578.308326] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14580.192326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14580.197057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14580.206742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14580.215392] Lustre: Skipped 1 previous similar message [14580.232990] Lustre: Skipped 1 previous similar message [14585.318704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14585.330702] Lustre: Skipped 1 previous similar message [14586.459535] Lustre: server umount lustre-OST0000 complete [14590.438149] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14590.442275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14590.447236] Lustre: Skipped 1 previous similar message [14590.457904] Lustre: Skipped 1 previous similar message [14595.554407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14595.562082] Lustre: Skipped 1 previous similar message [14596.883727] Lustre: server umount lustre-MDT0000 complete [14600.868515] LustreError: 522867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782255023 with bad export cookie 4808435751859940730 [14600.888043] LustreError: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14600.890204] LustreError: 522867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14601.236831] Lustre: server umount lustre-MDT0001 complete [14627.358990] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_hostid [14636.642869] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing load_modules_local [14665.165701] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14420 sec ========= 18:51:25 (1782255085) [14666.962029] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 18:51:27 (1782255087) === [14670.611571] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 18:51:31 (1782255091) === [14709.537789] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing unload_modules_local [14712.649910] Key type lgssc unregistered [14713.106693] LNet: 531486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14713.122543] LNetError: 531486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [14713.145234] LNet: Removed LNI 192.168.203.145@tcp [14714.098126] Key type .llcrypt unregistered [14714.101430] Key type ._llcrypt unregistered