[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 478595859 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2448MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003297] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011128] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013046] SELinux: Initializing. [ 0.014052] *** VALIDATE selinux *** [ 0.022115] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026774] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028067] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030072] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031114] *** VALIDATE tmpfs *** [ 0.033114] *** VALIDATE proc *** [ 0.034253] *** VALIDATE cgroup *** [ 0.035012] *** VALIDATE cgroup2 *** [ 0.036291] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039030] Spectre V2 : User space: Vulnerable [ 0.040010] Speculative Store Bypass: Vulnerable [ 0.042554] debug: unmapping init [mem 0xffffffff99659000-0xffffffff99660fff] [ 0.044178] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045676] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046025] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049013] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051015] ... fixed-purpose events: 3 [ 0.052012] ... event mask: 000000070000000f [ 0.053294] rcu: Hierarchical SRCU implementation. [ 0.055410] smp: Bringing up secondary CPUs ... [ 0.056603] x86: Booting SMP configuration: [ 0.057026] .... node #0, CPUs: #1 #2 #3 [ 0.060347] smp: Brought up 1 node, 4 CPUs [ 0.062015] smpboot: Max logical packages: 1 [ 0.063012] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.231028] node 0 deferred pages initialised in 164ms [ 0.234438] devtmpfs: initialized [ 0.235276] x86/mm: Memory block size: 128MB [ 0.238000] gcov: version magic: 0x41383552 [ 0.240263] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.243077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.246295] pinctrl core: initialized pinctrl subsystem [ 0.248229] [ 0.248838] ************************************************************* [ 0.252016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254013] ** ** [ 0.256011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.258011] ** ** [ 0.260012] ** This means that this kernel is built to expose internal ** [ 0.262011] ** IOMMU data structures, which may compromise security on ** [ 0.264028] ** your system. ** [ 0.265008] ** ** [ 0.267010] ** If you see this message and you are not debugging the ** [ 0.269012] ** kernel, report this immediately to your vendor! ** [ 0.271070] ** ** [ 0.273014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275013] ************************************************************* [ 0.277686] NET: Registered protocol family 16 [ 0.280714] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.283123] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.285055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.289095] cpuidle: using governor menu [ 0.290662] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.293026] PCI: Using configuration type 1 for base access [ 0.295187] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.305104] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.308032] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.311084] cryptd: max_cpu_qlen set to 1000 [ 0.314281] ACPI: Added _OSI(Module Device) [ 0.316017] ACPI: Added _OSI(Processor Device) [ 0.317016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.319017] ACPI: Added _OSI(Processor Aggregator Device) [ 0.324406] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.329501] ACPI: Interpreter enabled [ 0.331090] ACPI: PM: (supports S0 S3 S4 S5) [ 0.333019] ACPI: Using IOAPIC for interrupt routing [ 0.335256] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.339421] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.351034] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.353063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.357029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.360125] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.365535] acpiphp: Slot [2] registered [ 0.367134] acpiphp: Slot [5] registered [ 0.369115] acpiphp: Slot [6] registered [ 0.370219] acpiphp: Slot [7] registered [ 0.372156] acpiphp: Slot [8] registered [ 0.373144] acpiphp: Slot [9] registered [ 0.375086] acpiphp: Slot [10] registered [ 0.376122] acpiphp: Slot [3] registered [ 0.378146] acpiphp: Slot [4] registered [ 0.379135] acpiphp: Slot [11] registered [ 0.381175] acpiphp: Slot [12] registered [ 0.383140] acpiphp: Slot [13] registered [ 0.384106] acpiphp: Slot [14] registered [ 0.386164] acpiphp: Slot [15] registered [ 0.388181] acpiphp: Slot [16] registered [ 0.390226] acpiphp: Slot [17] registered [ 0.392169] acpiphp: Slot [18] registered [ 0.394195] acpiphp: Slot [19] registered [ 0.396109] acpiphp: Slot [20] registered [ 0.398217] acpiphp: Slot [21] registered [ 0.399093] acpiphp: Slot [22] registered [ 0.400117] acpiphp: Slot [23] registered [ 0.401138] acpiphp: Slot [24] registered [ 0.403081] acpiphp: Slot [25] registered [ 0.404089] acpiphp: Slot [26] registered [ 0.405137] acpiphp: Slot [27] registered [ 0.407101] acpiphp: Slot [28] registered [ 0.408146] acpiphp: Slot [29] registered [ 0.410114] acpiphp: Slot [30] registered [ 0.411134] acpiphp: Slot [31] registered [ 0.413060] PCI host bridge to bus 0000:00 [ 0.414015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.416023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.418028] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.421022] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.423023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.426038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.428250] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.432341] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.436635] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.447023] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.452000] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.455029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.458027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.461035] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.466371] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.469896] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.473042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.475964] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.481020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.493018] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.497015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.505262] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.514022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.521020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.540022] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.551528] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.560016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.569017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.597015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.612019] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.622025] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.634024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.673021] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.685556] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.696025] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.703023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.729020] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.743635] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.754016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.761015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.781020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.795452] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.802023] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.809020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.827022] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.841422] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.843351] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.845321] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.849493] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.852318] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.857145] iommu: Default domain type: Passthrough [ 0.859652] SCSI subsystem initialized [ 0.861201] ACPI: bus type USB registered [ 0.863196] usbcore: registered new interface driver usbfs [ 0.866134] usbcore: registered new interface driver hub [ 0.868135] usbcore: registered new device driver usb [ 0.870251] pps_core: LinuxPPS API ver. 1 registered [ 0.872018] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.876087] PTP clock support registered [ 0.878095] EDAC MC: Ver: 3.0.0 [ 0.879272] PCI: Using ACPI for IRQ routing [ 0.881122] NetLabel: Initializing [ 0.882011] NetLabel: domain hash size = 128 [ 0.884016] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.886164] NetLabel: unlabeled traffic allowed by default [ 0.889235] vgaarb: loaded [ 0.891461] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.892019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.896210] clocksource: Switched to clocksource kvm-clock [ 1.006815] VFS: Disk quotas dquot_6.6.0 [ 1.008735] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.011670] *** VALIDATE ramfs *** [ 1.013136] *** VALIDATE hugetlbfs *** [ 1.015190] pnp: PnP ACPI init [ 1.018184] pnp: PnP ACPI: found 6 devices [ 1.038157] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.043716] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.046520] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.049026] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.051807] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.054750] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.058184] NET: Registered protocol family 2 [ 1.061419] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.067769] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.072172] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.080969] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.086408] TCP: Hash tables configured (established 65536 bind 65536) [ 1.091374] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.096422] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.100579] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.104996] NET: Registered protocol family 1 [ 1.108556] RPC: Registered named UNIX socket transport module. [ 1.110897] RPC: Registered udp transport module. [ 1.113138] RPC: Registered tcp transport module. [ 1.115390] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.118781] NET: Registered protocol family 44 [ 1.120924] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.123644] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.126291] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.129317] PCI: CLS 0 bytes, default 64 [ 1.131553] Unpacking initramfs... [ 2.568760] debug: unmapping init [mem 0xffff93cfbcc54000-0xffff93cfbffbffff] [ 2.573504] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.576135] software IO TLB: mapped [mem 0x00000000b8c54000-0x00000000bcc54000] (64MB) [ 2.579522] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.080952] Initialise system trusted keyrings [ 3.082924] Key type blacklist registered [ 3.085086] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.109495] zbud: loaded [ 3.116333] *** VALIDATE nfs *** [ 3.117554] *** VALIDATE nfs4 *** [ 3.120444] pstore: using deflate compression [ 3.123775] Platform Keyring initialized [ 3.237345] NET: Registered protocol family 38 [ 3.239618] Key type asymmetric registered [ 3.241142] Asymmetric key parser 'x509' registered [ 3.243258] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.247391] io scheduler mq-deadline registered [ 3.249139] io scheduler kyber registered [ 3.251197] io scheduler bfq registered [ 3.252978] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.255933] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.258624] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.261293] ACPI: Power Button [PWRF] [ 3.268391] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.275436] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.289711] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.299234] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.318616] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.347422] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.377153] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.382734] Non-volatile memory driver v1.3 [ 3.384832] Linux agpgart interface v0.103 [ 3.421240] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.424095] vda: detected capacity change from 0 to 74760192 [ 3.445836] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.449135] vdb: detected capacity change from 0 to 1073741824 [ 3.465354] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.468591] vdc: detected capacity change from 0 to 2621440000 [ 3.485102] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.488111] vdd: detected capacity change from 0 to 2621440000 [ 3.502773] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.505505] vde: detected capacity change from 0 to 4294967296 [ 3.520690] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.523979] vdf: detected capacity change from 0 to 4294967296 [ 3.531421] libphy: Fixed MDIO Bus: probed [ 3.537071] usbcore: registered new interface driver usbserial_generic [ 3.540161] usbserial: USB Serial support registered for generic [ 3.542672] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.548858] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.550924] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.554077] mousedev: PS/2 mouse device common for all mice [ 3.557068] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.558740] rtc_cmos 00:05: RTC can wake from S4 [ 3.564251] rtc_cmos 00:05: registered as rtc0 [ 3.565126] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.566791] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.573115] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.573857] intel_pstate: CPU model not supported [ 3.580644] hid: raw HID events driver (C) Jiri Kosina [ 3.584132] usbcore: registered new interface driver usbhid [ 3.586830] usbhid: USB HID core driver [ 3.589025] drop_monitor: Initializing network drop monitor service [ 3.592338] Initializing XFRM netlink socket [ 3.595130] NET: Registered protocol family 10 [ 3.598693] Segment Routing with IPv6 [ 3.600535] NET: Registered protocol family 17 [ 3.603315] mpls_gso: MPLS GSO support [ 3.609907] RAS: Correctable Errors collector initialized. [ 3.612137] AVX version of gcm_enc/dec engaged. [ 3.613783] AES CTR mode by8 optimization enabled [ 3.693160] sched_clock: Marking stable (3693133030, 0)->(4631336892, -938203862) [ 3.696429] registered taskstats version 1 [ 3.698369] Loading compiled-in X.509 certificates [ 3.700504] zswap: loaded using pool lzo/zbud [ 3.755094] Key type big_key registered [ 3.768962] Key type encrypted registered [ 3.771207] ima: No TPM chip found, activating TPM-bypass! [ 3.773772] ima: Allocated hash algorithm: sha1 [ 3.775861] ima: No architecture policies found [ 3.778099] evm: Initialising EVM extended attributes: [ 3.780301] evm: security.selinux [ 3.781925] evm: security.ima [ 3.783857] evm: security.capability [ 3.785667] evm: HMAC attrs: 0x1 [ 3.788541] rtc_cmos 00:05: setting system clock to 2026-08-31 05:10:12 UTC (1788153012) [ 3.794838] debug: unmapping init [mem 0xffffffff9a603000-0xffffffff9a7fffff] [ 3.798721] debug: unmapping init [mem 0xffffffff99382000-0xffffffff99658fff] [ 3.811469] Write protecting the kernel read-only data: 28672k [ 3.815375] debug: unmapping init [mem 0xffffffff97a03000-0xffffffff97bfffff] [ 3.819119] debug: unmapping init [mem 0xffffffff98314000-0xffffffff983fffff] [ 3.863522] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.874549] systemd[1]: Detected virtualization kvm. [ 3.878073] systemd[1]: Detected architecture x86-64. [ 3.881350] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.912366] systemd[1]: No hostname configured. [ 3.915324] systemd[1]: Set hostname to . [ 3.918732] random: systemd: uninitialized urandom read (16 bytes read) [ 3.922952] systemd[1]: Initializing machine ID from random generator. [ 3.975348] random: ln: uninitialized urandom read (6 bytes read) [ 4.067108] random: systemd: uninitialized urandom read (16 bytes read) [ 4.070259] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.075232] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 4.086841] systemd[1]: Started Memstrack Anylazing Service. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Apply Kernel Variables... [ OK ] Reached target Timers. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.777416] device-mapper: uevent: version 1.0.3 [ 4.779655] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.503686] random: fast init done [ 5.541900] virtio_net virtio0 ens2: renamed from eth0 [ 5.681825] scsi host0: ata_piix [ 5.785273] scsi host1: ata_piix [ 5.787246] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.790173] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.079763] dracut-initqueue[586]: RTNETLINK answers: File exists [ 10.295183] random: crng init done [ 10.296895] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.825152] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.050691] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.347406] SELinux: Disabled at runtime. [ 12.412665] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.423313] systemd[1]: Detected virtualization kvm. [ 12.425184] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.985186] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.989416] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.993758] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.997580] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.000523] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.008153] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.015156] systemd[1]: Mounting POSIX Message Queue File System... Mounting POSIX Message Queue File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Switch Root. Starting Create list of required st…ce nodes for the current kernel... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on[ 13.079950] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS udev Kernel Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Mounted /mnt. [ 13.457277] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.740790] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.758349] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.917132] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.926520] EDAC sbridge: Ver: 1.1.2 [ 15.653975] Key type dns_resolver registered [ 15.958085] NFS: Registering the id_resolver key type [ 15.960335] Key type id_resolver registered [ 15.962169] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg307-server login: [ 42.616077] libcfs: loading out-of-tree module taints kernel. [ 42.636099] Key type ._llcrypt registered [ 42.637639] Key type .llcrypt registered [ 42.687906] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 53.780552] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 55.626737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 55.637500] alg: No test for adler32 (adler32-zlib) [ 57.009587] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 57.776550] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 59.503254] Key type lgssc registered [ 61.192327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 81.668073] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.217826] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 141.537291] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 141.576314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 142.958937] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 142.989710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.105414] Lustre: lustre-MDT0000: new disk, initializing [ 143.202734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.253794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 147.744895] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 160.597521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 160.741941] Lustre: 6483:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 160.773568] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 160.777650] Lustre: Skipped 1 previous similar message [ 160.892509] Lustre: lustre-MDT0001: new disk, initializing [ 160.986380] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 161.044519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 161.068840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 165.142484] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.155337] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 180.533143] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 180.868155] Lustre: lustre-OST0000: new disk, initializing [ 180.874790] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 180.879731] Lustre: 8422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 180.960472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 186.904259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 186.927704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 186.995791] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 187.347571] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 203.056793] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 203.241594] Lustre: lustre-OST0001: new disk, initializing [ 203.245446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 203.259904] Lustre: 9494:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 203.356788] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 208.955088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 208.965530] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 209.031501] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 210.747761] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 224.143697] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.469762] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 240.018446] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing check_logdir /tmp/testlogs/ [ 245.955537] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing yml_node [ 251.327490] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 254.379731] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 254.473480] hrtimer: interrupt took 1578829 ns [ 257.213187] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 259.109442] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 01:14:26 EDT 2026 [ 278.711981] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 280.784903] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 299.488308] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 299.496570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 299.507117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.028636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 301.029557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.055817] Lustre: Skipped 1 previous similar message [ 301.068646] Lustre: Skipped 3 previous similar messages [ 304.089528] Lustre: server umount lustre-MDT0000 complete [ 308.022322] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153316 with bad export cookie 16603046812416713718 [ 308.023410] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 308.034175] LustreError: 6476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 311.263761] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.265135] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.272316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.276572] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 311.334453] Lustre: Skipped 2 previous similar messages [ 313.321772] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 313.322485] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 313.346787] LustreError: 6490:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 313.351144] Lustre: Skipped 2 previous similar messages [ 314.461989] Lustre: server umount lustre-MDT0001 complete [ 318.271483] Lustre: server umount lustre-OST0000 complete [ 322.157414] Lustre: server umount lustre-OST0001 complete [ 329.220755] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 339.006471] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 378.228559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 378.519749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 378.567190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 378.686649] Lustre: lustre-MDT0000: new disk, initializing [ 378.776770] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 378.797945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 383.936434] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 395.351519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 395.450452] Lustre: 15049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 395.510756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 395.515375] Lustre: Skipped 1 previous similar message [ 395.583302] Lustre: lustre-MDT0001: new disk, initializing [ 395.686207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 395.796929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 395.828057] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 400.217778] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 411.309750] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 418.348286] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 425.929274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 426.245308] Lustre: lustre-OST0000: new disk, initializing [ 426.250643] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 426.257469] Lustre: 17020:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 426.365304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 427.438128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 427.460062] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 427.704030] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 433.822285] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 444.849708] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 449.164344] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 449.377062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 454.425859] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 454.646489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 458.210129] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 458.213874] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 458.224047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 458.228817] Lustre: Skipped 1 previous similar message [ 462.984319] Lustre: server umount lustre-OST0000 complete [ 467.427511] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 467.436456] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 467.452683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 467.465511] Lustre: Skipped 1 previous similar message [ 473.000966] Lustre: server umount lustre-MDT0000 complete [ 473.570229] LustreError: 15058:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 473.589683] LustreError: 15058:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 476.853403] LustreError: 15847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153485 with bad export cookie 16603046812416715244 [ 476.858854] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 476.863496] LustreError: 15847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 477.274945] Lustre: server umount lustre-MDT0001 complete [ 482.091687] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:18:09 (1788153489) [ 484.987644] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 486.718943] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:18:14 (1788153494) [ 493.615561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 494.176933] LustreError: 19429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 494.269462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.917768] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 499.694991] LustreError: 19430:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 507.799800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 512.671995] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 520.498561] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 526.335333] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 533.715634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 540.817775] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 549.139583] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 560.103807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 560.119121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 560.147049] Lustre: Skipped 1 previous similar message [ 560.155075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 560.161993] Lustre: Skipped 2 previous similar messages [ 566.035376] Lustre: server umount lustre-OST0000 complete [ 576.542587] Lustre: server umount lustre-MDT0000 complete [ 578.531092] LustreError: 19429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 578.559420] LustreError: 19429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 580.936827] LustreError: 19410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153589 with bad export cookie 16603046812416716133 [ 580.942901] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 580.958246] LustreError: 19410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 581.384917] Lustre: server umount lustre-MDT0001 complete [ 592.936775] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 595.742810] Key type lgssc unregistered [ 596.004854] LNet: 22964:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 596.013074] LNetError: 22964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 597.029781] LNet: Removed LNI 192.168.203.107@tcp [ 598.140175] Key type .llcrypt unregistered [ 598.144745] Key type ._llcrypt unregistered [ 623.115402] Key type ._llcrypt registered [ 623.118501] Key type .llcrypt registered [ 623.222714] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 624.201842] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 624.344149] alg: No test for adler32 (adler32-zlib) [ 625.623077] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 625.962311] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 627.647894] Key type lgssc registered [ 629.160681] Lustre: Echo OBD driver; http://www.lustre.org/ [ 638.835942] loop: module loaded [ 638.954699] loop0: detected capacity change from 0 to 204800000 [ 639.142535] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.163574] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 639.609935] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.976915] loop0: detected capacity change from 0 to 204800000 [ 646.143629] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 646.180758] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 646.201366] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 651.985134] loop0: detected capacity change from 0 to 204800000 [ 658.347161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 658.372167] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.660218] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 659.722106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 659.826262] Lustre: lustre-MDT0000: new disk, initializing [ 659.971878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 660.034260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 665.167289] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 675.103043] loop1: detected capacity change from 0 to 204800000 [ 681.125179] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 681.303773] Lustre: lustre-OST07c6: new disk, initializing [ 681.308509] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 681.313363] Lustre: Skipped 1 previous similar message [ 681.317560] Lustre: 26153:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 681.389661] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 686.689995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 686.695895] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 686.724276] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 688.045560] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 693.092438] Lustre: 25140:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 693.104111] Lustre: 25140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 693.110823] Lustre: 25140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 693.118841] Lustre: 25140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 693.126927] Lustre: 25140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 693.135039] Lustre: 25140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 712.145262] Lustre: 25139:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 712.159066] Lustre: 25139:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 712.169264] Lustre: 25139:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 712.175301] Lustre: 25139:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 712.185214] Lustre: 25139:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 712.192238] Lustre: 25139:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 717.699317] Lustre: 26161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 717.710215] Lustre: 26161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 717.716428] Lustre: 26161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 717.724280] Lustre: 26161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.732261] Lustre: 26161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.741486] Lustre: 26161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.758461] Lustre: 26161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 717.768198] Lustre: 26161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.775327] Lustre: 26161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 717.783303] Lustre: 26161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 717.801215] Lustre: 26161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 717.807300] Lustre: 26161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.109432] Lustre: 25141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 723.115902] Lustre: 25141:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 723.127323] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 723.133541] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.152085] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 723.170572] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.189081] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 723.200622] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.211559] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 723.221649] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 723.229124] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 723.232882] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 729.642967] Lustre: 26161:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 729.653869] Lustre: 26161:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 729.663992] Lustre: 26161:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 729.677134] Lustre: 26161:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 729.689888] Lustre: 26161:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 729.700466] Lustre: 26161:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 729.708069] Lustre: 26161:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 729.713522] Lustre: 26161:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 729.720089] Lustre: 26161:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 729.727027] Lustre: 26161:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 729.732657] Lustre: 26161:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 729.738196] Lustre: 26161:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 737.709660] Lustre: 25141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 737.725687] Lustre: 25141:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 737.736123] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 737.744568] Lustre: 25141:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 737.752062] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 737.766190] Lustre: 25141:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 737.778623] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 737.789405] Lustre: 25141:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 737.794766] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 737.798900] Lustre: 25141:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 737.807092] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 737.815888] Lustre: 25141:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 788.962388] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 788.986194] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 789.001081] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 793.400299] Lustre: server umount lustre-OST07c6 complete [ 804.443876] Lustre: server umount lustre-MDT0000 complete [ 822.497938] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 825.993710] Key type lgssc unregistered [ 826.346563] LNet: 28714:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.364378] LNetError: 28714:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.399864] LNet: Removed LNI 192.168.203.107@tcp [ 827.391403] Key type .llcrypt unregistered [ 827.396197] Key type ._llcrypt unregistered [ 849.322482] Key type ._llcrypt registered [ 849.330711] Key type .llcrypt registered [ 849.515861] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 851.303434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 851.342145] alg: No test for adler32 (adler32-zlib) [ 852.459991] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 852.662789] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 854.399193] Key type lgssc registered [ 856.483699] Lustre: Echo OBD driver; http://www.lustre.org/ [ 865.915626] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 865.928311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 867.232632] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.742935] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 876.527935] LustreError: 29987:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 876.571728] LustreError: 29987:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 881.639556] LustreError: 30006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 884.598764] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 885.298740] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 892.246523] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 900.009515] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 905.942403] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 912.836351] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.130809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 918.818339] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 927.741348] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 936.637090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 955.362495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 955.377762] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 955.407396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 957.414514] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 957.420713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 957.421940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 957.439169] Lustre: Skipped 1 previous similar message [ 958.583574] Lustre: server umount lustre-OST0000 complete [ 962.539995] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.561811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 967.136839] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 967.147590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 967.163983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 968.580046] Lustre: server umount lustre-MDT0000 complete [ 971.871622] LustreError: 29967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153980 with bad export cookie 10332413151703976127 [ 971.875060] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 971.880026] LustreError: 29967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 972.394228] Lustre: server umount lustre-MDT0001 complete [ 985.159511] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 988.593427] Key type lgssc unregistered [ 988.972692] LNet: 33749:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 988.978897] LNetError: 33749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 989.001808] LNet: Removed LNI 192.168.203.107@tcp [ 989.999784] Key type .llcrypt unregistered [ 990.001800] Key type ._llcrypt unregistered [ 1011.813155] Key type ._llcrypt registered [ 1011.818843] Key type .llcrypt registered [ 1011.973679] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 1013.035869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1013.067424] alg: No test for adler32 (adler32-zlib) [ 1014.169633] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1014.428916] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 1016.111202] Key type lgssc registered [ 1017.582391] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1026.476827] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1026.523768] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1028.038337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1033.029804] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1038.305152] LustreError: 35022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.320561] LustreError: 35022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1042.646923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1043.210586] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1048.575561] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1056.594379] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1063.777311] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1072.434581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1072.840127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1073.915891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1080.191338] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1088.365111] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1103.840311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1103.847448] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.851286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1103.869983] Lustre: Skipped 1 previous similar message [ 1107.401704] Lustre: server umount lustre-OST0000 complete [ 1114.091306] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.105621] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.122634] Lustre: Skipped 2 previous similar messages [ 1117.806919] Lustre: server umount lustre-MDT0000 complete [ 1119.212550] LustreError: 35021:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1119.248130] LustreError: 35021:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1122.431536] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154131 with bad export cookie 6316241678350728043 [ 1122.440363] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1122.440437] LustreError: 35001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1122.765660] Lustre: server umount lustre-MDT0001 complete [ 1135.546182] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 1138.883879] Key type lgssc unregistered [ 1139.299734] LNet: 38579:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1139.309180] LNetError: 38579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1139.326706] LNet: Removed LNI 192.168.203.107@tcp [ 1140.367270] Key type .llcrypt unregistered [ 1140.369317] Key type ._llcrypt unregistered [ 1169.382744] Key type ._llcrypt registered [ 1169.390458] Key type .llcrypt registered [ 1169.538818] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 1171.430806] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1171.452244] alg: No test for adler32 (adler32-zlib) [ 1172.866355] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1173.352394] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 1175.224069] Key type lgssc registered [ 1176.646107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1187.881356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1187.896670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1189.331371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1193.301406] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1199.589506] LustreError: 39998:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1199.609994] LustreError: 39998:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1202.586917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1202.911757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1207.717972] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1216.675898] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1223.397054] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1230.552163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1230.986238] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1235.135173] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1237.275034] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1245.865290] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1252.831870] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 1252.834501] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.852268] Lustre: Skipped 1 previous similar message [ 1252.903515] Lustre: Skipped 1 previous similar message [ 1257.920628] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 1257.942186] Lustre: Skipped 2 previous similar messages [ 1258.515636] Lustre: server umount lustre-MDT0000 complete [ 1263.039690] LustreError: 39993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1263.414771] LustreError: 39978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154272 with bad export cookie 2657497503952700039 [ 1263.423389] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1263.433265] LustreError: 39978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1268.157461] LustreError: 39993:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1268.169951] LustreError: 39993:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1268.187321] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [ 1268.193690] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1268.200710] Lustre: Skipped 3 previous similar messages [ 1268.225863] Lustre: Skipped 1 previous similar message [ 1270.035321] Lustre: server umount lustre-MDT0001 complete [ 1291.618683] Lustre: server umount lustre-OST0000 complete [ 1311.747463] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 1315.169817] Key type lgssc unregistered [ 1315.517431] LNet: 43858:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1315.535853] LNetError: 43858:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1316.593223] LNet: Removed LNI 192.168.203.107@tcp [ 1317.710124] Key type .llcrypt unregistered [ 1317.712712] Key type ._llcrypt unregistered [ 1345.554690] Key type ._llcrypt registered [ 1345.556394] Key type .llcrypt registered [ 1345.735569] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 1347.319278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1347.464213] alg: No test for adler32 (adler32-zlib) [ 1349.024874] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1349.447773] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 1351.217557] Key type lgssc registered [ 1352.825294] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1362.921959] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1362.932741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1364.543762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1369.181103] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1374.708830] LustreError: 45279:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1374.738033] LustreError: 45279:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1378.259921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1378.630264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1383.987126] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1393.227135] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1399.930369] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1408.113629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1408.455076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.458084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1414.718167] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1423.874396] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1432.042532] Lustre: lustre-OST0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 1434.080187] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1434.085964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1434.093077] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1434.099917] Lustre: Skipped 1 previous similar message [ 1434.106848] Lustre: Skipped 1 previous similar message [ 1437.128159] Lustre: lustre-OST0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 1437.743595] Lustre: server umount lustre-OST0000 complete [ 1458.664352] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1458.673403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1458.690655] Lustre: Skipped 1 previous similar message [ 1458.723225] Lustre: Skipped 1 previous similar message [ 1463.401157] Lustre: server umount lustre-MDT0000 complete [ 1463.776250] LustreError: 45280:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1463.802655] LustreError: 45280:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1468.005312] LustreError: 45260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154476 with bad export cookie 13614435813579386812 [ 1468.027127] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1468.029551] LustreError: 45260:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1474.711349] Lustre: server umount lustre-MDT0001 complete [ 1486.741784] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 1489.858192] Key type lgssc unregistered [ 1490.290315] LNet: 48990:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1490.304746] LNetError: 48990:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1490.329164] LNet: Removed LNI 192.168.203.107@tcp [ 1491.481527] Key type .llcrypt unregistered [ 1491.484829] Key type ._llcrypt unregistered [ 1519.170684] Key type ._llcrypt registered [ 1519.174958] Key type .llcrypt registered [ 1519.351245] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 1521.205922] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1521.278629] alg: No test for adler32 (adler32-zlib) [ 1522.457800] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1522.941886] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 1524.759145] Key type lgssc registered [ 1526.160168] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1535.851287] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1535.868038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1537.482927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1543.874437] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1547.754802] LustreError: 50408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1547.802855] LustreError: 50408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1552.865099] LustreError: 50409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1554.275515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.687731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1560.367874] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1569.140741] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1575.451807] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1584.970630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1585.590173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.067572] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1602.308332] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1608.134783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1612.007261] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1615.323195] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1627.736121] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-31 1h37m14s [ 1629.207137] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1631.712771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1631.722426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1631.739811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.805236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1632.817728] Lustre: Skipped 4 previous similar messages [ 1634.812257] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 1634.819070] Lustre: Skipped 1 previous similar message [ 1637.361935] Lustre: server umount lustre-MDT0000 complete [ 1638.901733] LustreError: 50404:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.926024] LustreError: 50404:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1640.824335] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1644.005031] LustreError: 50403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1644.046150] LustreError: 50403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1647.242091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1647.364206] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1647.617040] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1647.730207] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1648.626455] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1652.412375] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1656.013482] Lustre: DEBUG MARKER: Wait for df (36698) ... [ 1658.486593] Lustre: DEBUG MARKER: done [ 1663.974763] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.976452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1663.995453] Lustre: Skipped 3 previous similar messages [ 1669.092251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1669.098523] Lustre: Skipped 1 previous similar message [ 1669.293342] Lustre: server umount lustre-OST0000 complete [ 1674.209400] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1674.214253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1679.684715] Lustre: server umount lustre-MDT0000 complete [ 1680.366472] LustreError: 50408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1680.383866] LustreError: 50408:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 1683.633553] LustreError: 50389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154692 with bad export cookie 1915522917042791998 [ 1683.635770] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1683.664622] LustreError: 50389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1684.037657] Lustre: server umount lustre-MDT0001 complete [ 1695.769557] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 1698.722867] Key type lgssc unregistered [ 1699.066755] LNet: 55335:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1699.072638] LNetError: 55335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1699.093148] LNet: Removed LNI 192.168.203.107@tcp [ 1699.953124] Key type .llcrypt unregistered [ 1699.955217] Key type ._llcrypt unregistered [ 1742.221853] Key type ._llcrypt registered [ 1742.225530] Key type .llcrypt registered [ 1742.356777] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 1743.515841] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1743.612769] alg: No test for adler32 (adler32-zlib) [ 1744.634673] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1744.889285] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 1746.591167] Key type lgssc registered [ 1747.770675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1755.493453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1755.521424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1756.875744] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1756.915320] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1757.032157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1762.018756] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1771.243927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1771.394586] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1771.423605] Lustre: 57548:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1771.777036] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1777.139471] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1785.128195] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1791.323188] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1798.255877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1798.495583] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1798.702235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1804.889807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1805.158978] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 1812.833483] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1820.072537] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-31 1h40m27s [ 1821.640201] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1829.641577] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1832.537847] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1832.598361] LustreError: 57553:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1852.928449] Lustre: lustre-MDT0000: Client 5280f771-5d69-46b9-95c5-50b76c2a1bb7 (at 192.168.203.7@tcp) reconnecting [ 1872.639302] LustreError: 57553:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1875.254050] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1878.411206] Lustre: DEBUG MARKER: done [ 1886.691854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1886.709843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1886.729751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.740491] Lustre: Skipped 1 previous similar message [ 1891.814898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1891.821995] Lustre: Skipped 1 previous similar message [ 1896.929292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1896.935714] Lustre: Skipped 1 previous similar message [ 1899.999176] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1900.195551] Lustre: server umount lustre-OST0000 complete [ 1905.124535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1905.135183] LustreError: Skipped 1 previous similar message [ 1905.140889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1905.157559] Lustre: Skipped 1 previous similar message [ 1905.167391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1910.672663] Lustre: server umount lustre-MDT0000 complete [ 1915.579449] LustreError: 57538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154924 with bad export cookie 1686956257327330213 [ 1915.589384] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1915.595396] LustreError: 57538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1916.056900] Lustre: server umount lustre-MDT0001 complete [ 1927.975150] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 1931.331623] Key type lgssc unregistered [ 1931.662509] LNet: 61878:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1931.677937] LNetError: 61878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1931.710158] LNet: Removed LNI 192.168.203.107@tcp [ 1932.746231] Key type .llcrypt unregistered [ 1932.749624] Key type ._llcrypt unregistered [ 1982.177195] Key type ._llcrypt registered [ 1982.184934] Key type .llcrypt registered [ 1982.336574] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 1983.842385] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1984.004491] alg: No test for adler32 (adler32-zlib) [ 1985.345336] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1985.645482] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 1987.295196] Key type lgssc registered [ 1988.658800] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1992.277550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1992.291500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.534624] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1993.549638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1993.702421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1994.803630] Lustre: Failing over lustre-MDT0000 [ 1995.380544] Lustre: server umount lustre-MDT0000 complete [ 2002.352549] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:43:30 (1788155010) [ 2011.363796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2011.958793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.033529] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2027.255988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2027.326205] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2027.345857] Lustre: 64681:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2027.561761] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2032.122500] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2039.460781] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2046.294378] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2053.384217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2053.623780] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2053.887497] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2059.389839] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2059.901558] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2068.378493] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2076.697900] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2085.346973] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2085.350196] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2085.359650] Lustre: Skipped 2 previous similar messages [ 2085.376990] Lustre: Skipped 1 previous similar message [ 2090.466771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2090.473919] Lustre: Skipped 2 previous similar messages [ 2095.590395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.143437] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2098.421977] Lustre: server umount lustre-MDT0000 complete [ 2102.734701] LustreError: 64676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155111 with bad export cookie 13545929291418906100 [ 2102.748874] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2102.750859] LustreError: 64676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2103.097476] Lustre: server umount lustre-MDT0001 complete [ 2107.285516] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2112.659659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2121.534870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.143117] Lustre: 63606:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155114/real 1788155114] req@00000000b263b703 x1875016411031808/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155130 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2122.174255] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2124.259251] Lustre: 63608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155116/real 1788155116] req@00000000d62bfbd5 x1875016411031936/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2126.377500] Lustre: 63608:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155119/real 1788155119] req@0000000062b38702 x1875016411032064/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155135 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2127.543828] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2127.560092] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2127.624190] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2127.703962] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2128.607421] Lustre: 63606:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155121/real 1788155121] req@00000000e3356a3a x1875016411032320/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2130.725890] LustreError: 66533:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2130.746347] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2131.817931] LustreError: 68460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2131.848038] LustreError: 68460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2133.107379] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2133.987205] Lustre: 63607:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155126/real 1788155126] req@00000000c7fddec4 x1875016411032448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155142 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2137.061117] LustreError: 68459:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2137.096785] LustreError: 68459:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2142.089874] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.182258] LustreError: 68460:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2142.206631] LustreError: 68460:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2142.554977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2147.033986] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2147.826255] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2155.109514] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2160.816878] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2168.288797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2168.294166] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.297848] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.312544] Lustre: Skipped 2 previous similar messages [ 2168.344101] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2169.987831] Lustre: server umount lustre-MDT0000 complete [ 2174.208595] LustreError: 66539:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155182 with bad export cookie 13545929291418908382 [ 2174.225919] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2174.723981] Lustre: server umount lustre-MDT0001 complete [ 2181.669336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2192.218314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.911596] Lustre: 63606:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788155187/real 1788155187] req@0000000076b53dda x1875016411059072/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788155203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2194.940860] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.071926] LustreError: 63605:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e15c6683 x1875016411059968/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2199.583094] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.623695] LustreError: 66532:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2199.709826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2199.809558] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2205.052390] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2213.444105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2218.995569] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2219.323073] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2224.113768] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2229.832689] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2236.480089] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2244.581215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2244.587029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2244.596236] Lustre: Skipped 1 previous similar message [ 2244.609060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2244.617704] Lustre: Skipped 4 previous similar messages [ 2249.698823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2249.716794] Lustre: Skipped 2 previous similar messages [ 2258.399258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2258.664818] Lustre: server umount lustre-MDT0000 complete [ 2259.942630] LustreError: 72103:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2259.963170] LustreError: 72103:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2262.710264] LustreError: 64676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155271 with bad export cookie 13545929291418909761 [ 2262.723812] LustreError: 64676:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 2262.726025] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2263.050439] Lustre: server umount lustre-MDT0001 complete [ 2269.762643] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2273.509028] Lustre: DEBUG MARKER: files compared the same [ 2286.320623] Lustre: server umount lustre-OST0000 complete [ 2305.854032] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 2310.314103] Key type lgssc unregistered [ 2310.522227] LNet: 74418:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2310.537993] LNetError: 74418:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2310.561888] LNet: Removed LNI 192.168.203.107@tcp [ 2311.597202] Key type .llcrypt unregistered [ 2311.600145] Key type ._llcrypt unregistered [ 2336.350583] Key type ._llcrypt registered [ 2336.352834] Key type .llcrypt registered [ 2336.465810] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 2337.802261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2337.894314] alg: No test for adler32 (adler32-zlib) [ 2338.937352] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2339.131104] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 2340.791215] Key type lgssc registered [ 2341.885287] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2349.085949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2349.108088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2350.645688] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2354.985688] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug +malloc all [ 2360.804457] LustreError: 75834:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2360.828032] LustreError: 75834:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2363.520094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.929504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2368.269260] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug +malloc all [ 2376.196548] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2382.417705] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2388.514956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2388.853626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2395.722573] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug +malloc all [ 2396.017764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2404.490075] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2419.681063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.684051] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2419.711314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2419.726399] Lustre: Skipped 1 previous similar message [ 2422.935657] Lustre: server umount lustre-OST0000 complete [ 2429.929060] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2429.940389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2429.945554] Lustre: Skipped 2 previous similar messages [ 2429.960768] Lustre: Skipped 1 previous similar message [ 2434.521622] Lustre: server umount lustre-MDT0000 complete [ 2435.039961] LustreError: 75829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2435.060680] LustreError: 75829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2437.803369] LustreError: 75814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155446 with bad export cookie 1536910039796479628 [ 2437.811531] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2437.817350] LustreError: 75814:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2438.178548] Lustre: server umount lustre-MDT0001 complete [ 2447.581773] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 2450.681876] Key type lgssc unregistered [ 2450.976353] LNet: 79385:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2450.988939] LNetError: 79385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2452.008974] LNet: Removed LNI 192.168.203.107@tcp [ 2452.793229] Key type .llcrypt unregistered [ 2452.796943] Key type ._llcrypt unregistered [ 2473.599050] Key type ._llcrypt registered [ 2473.601077] Key type .llcrypt registered [ 2473.721265] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 2474.708867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2474.909792] alg: No test for adler32 (adler32-zlib) [ 2476.040921] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2476.361774] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 2478.047549] Key type lgssc registered [ 2479.359782] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2485.950029] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2485.995219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2501.727447] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2506.847802] LustreError: 80768:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.107@tcp: failed processing log, type 4: rc = -110 [ 2533.535404] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2539.154134] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2546.555891] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2553.995534] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2554.246393] Lustre: *** cfs_fail_loc=706, val=0*** [ 2554.492672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2554.580423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2558.796532] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2559.969863] LustreError: 80792:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2559.986988] LustreError: 80792:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2565.095424] LustreError: 80793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.164926] LustreError: 81852:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2568.180844] LustreError: 81852:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2568.239169] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2568.495690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2572.728371] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2581.451668] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2588.400901] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2594.280213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2594.287085] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2594.304429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2594.319908] Lustre: Skipped 1 previous similar message [ 2597.109399] Lustre: server umount lustre-OST0000 complete [ 2604.512626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2604.529345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2604.540472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2604.555879] Lustre: Skipped 2 previous similar messages [ 2604.566181] Lustre: Skipped 1 previous similar message [ 2608.108053] Lustre: server umount lustre-MDT0000 complete [ 2609.638373] LustreError: 81857:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2609.664071] LustreError: 81857:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2612.842040] LustreError: 80776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155621 with bad export cookie 841703692350667094 [ 2612.844202] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2612.855695] LustreError: 80776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2613.165317] Lustre: server umount lustre-MDT0001 complete [ 2628.536376] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 2632.089755] Key type lgssc unregistered [ 2632.457363] LNet: 84384:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2632.482075] LNetError: 84384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2632.507182] LNet: Removed LNI 192.168.203.107@tcp [ 2633.711490] Key type .llcrypt unregistered [ 2633.714077] Key type ._llcrypt unregistered [ 2661.403552] Key type ._llcrypt registered [ 2661.406616] Key type .llcrypt registered [ 2661.513156] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 2662.760232] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2662.810389] alg: No test for adler32 (adler32-zlib) [ 2663.964522] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2664.243726] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 2665.954740] Key type lgssc registered [ 2667.292703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2675.302811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2675.336587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2680.742422] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2688.760442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2689.276822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2694.415228] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2694.631406] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.663488] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2698.729324] LustreError: 86499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.840896] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.888338] LustreError: 86496:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2704.391975] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2704.793252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2709.988140] LustreError: 86500:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.027028] LustreError: 86500:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 2711.151435] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2714.087413] LustreError: 86499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.132689] LustreError: 86499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2719.248691] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2719.259276] Lustre: MGS: Client 06ca1d93-69cb-46d4-b954-39420d3f39c6 (at 0@lo) reconnecting [ 2719.275640] LustreError: 86510:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2719.511587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2719.599492] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2723.989524] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2736.608864] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2736.625202] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2736.640395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2740.197099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2740.202704] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.634291] Lustre: server umount lustre-OST0000 complete [ 2750.432220] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2750.438567] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2750.446977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2750.446985] Lustre: Skipped 1 previous similar message [ 2750.487747] Lustre: Skipped 1 previous similar message [ 2752.237101] Lustre: server umount lustre-MDT0000 complete [ 2755.554146] LustreError: 86499:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2755.581184] LustreError: 86499:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2756.422807] Lustre: server umount lustre-MDT0001 complete [ 2760.130899] Lustre: server umount MGS complete [ 2771.288073] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:56:19 (1788155779) [ 2801.017963] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 2811.079310] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 2846.184131] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2850.290629] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2859.700032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2859.880147] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2859.907989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2859.965554] Lustre: lustre-MDT0001: new disk, initializing [ 2860.026207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2860.064063] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2860.082750] Lustre: 93663:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2860.094314] LustreError: 93663:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2864.430835] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2874.488116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2874.737437] Lustre: lustre-OST0000: new disk, initializing [ 2874.740716] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2874.748667] Lustre: 94430:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2874.809913] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2881.047961] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2881.640975] Lustre: 94728:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000019651ffa x1875017121963776/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788155906 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2890.909606] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2898.799516] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2898.807657] Lustre: MGS: Client 1760c6b8-8d7c-4745-b239-b0492890c60a (at 0@lo) reconnecting [ 2898.839720] LustreError: 93649:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2898.876580] Lustre: 95475:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2898.972237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2899.088964] Lustre: lustre-MDT0000: new disk, initializing [ 2899.199611] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.216196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2899.256270] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2900.319938] Lustre: 95499:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000089cf872 x1875017121970816/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788155925 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2903.626644] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 2905.445685] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2909.164473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2909.179354] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2909.492967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 2934.753647] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2934.759635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2934.772599] Lustre: Skipped 1 previous similar message [ 2934.780146] Lustre: Skipped 1 previous similar message [ 2939.018576] Lustre: server umount lustre-OST0000 complete [ 2944.992723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2944.999818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2945.003895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2945.012773] Lustre: Skipped 1 previous similar message [ 2948.906213] Lustre: server umount lustre-MDT0000 complete [ 2950.120053] LustreError: 93635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2950.141443] LustreError: 93635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2959.531109] Lustre: server umount lustre-MDT0001 complete [ 2963.641134] Lustre: server umount MGS complete [ 2974.342692] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:59:41 (1788155981) [ 2993.688465] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 2996.457635] Key type lgssc unregistered [ 2996.786486] LNet: 97996:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2996.791359] LNetError: 97996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2997.861992] LNet: Removed LNI 192.168.203.107@tcp [ 2998.777602] Key type .llcrypt unregistered [ 2998.780349] Key type ._llcrypt unregistered [ 3017.119524] Key type ._llcrypt registered [ 3017.122760] Key type .llcrypt registered [ 3017.240223] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 3018.455066] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3018.542934] alg: No test for adler32 (adler32-zlib) [ 3019.597869] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3019.905097] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 3021.655259] Key type lgssc registered [ 3022.816901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3030.543237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3030.553599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3031.969560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.923089] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3042.274222] LustreError: 99220:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.299686] LustreError: 99220:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3045.894933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3046.313393] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3051.046971] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3059.315457] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3065.290722] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3073.931524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3074.372890] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.401178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3080.629935] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3089.328458] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3101.155504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3101.168428] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.189437] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3104.911521] Lustre: server umount lustre-OST0000 complete [ 3111.401961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3111.403424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3111.416731] Lustre: Skipped 2 previous similar messages [ 3111.434851] Lustre: Skipped 2 previous similar messages [ 3115.383560] Lustre: server umount lustre-MDT0000 complete [ 3116.517061] LustreError: 99221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3116.541392] LustreError: 99221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3119.361890] LustreError: 99199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156128 with bad export cookie 14037514842541091887 [ 3119.379106] LustreError: 99199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3119.380168] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3119.762572] Lustre: server umount lustre-MDT0001 complete [ 3132.150164] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 3135.262678] Key type lgssc unregistered [ 3135.579834] LNet: 102765:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3135.592409] LNetError: 102765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3135.619423] LNet: Removed LNI 192.168.203.107@tcp [ 3136.489267] Key type .llcrypt unregistered [ 3136.491402] Key type ._llcrypt unregistered [ 3152.457395] Key type ._llcrypt registered [ 3152.460384] Key type .llcrypt registered [ 3152.592609] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 3153.557169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3153.609134] alg: No test for adler32 (adler32-zlib) [ 3154.772309] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3155.002348] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 3156.671663] Key type lgssc registered [ 3157.877617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3163.952699] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3163.961049] LustreError: 103843:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3163.982640] LustreError: 103847:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3163.990492] LustreError: 103843:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4976 [ 3163.997744] LustreError: 103843:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3164.003549] LustreError: 103843:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3164.028687] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.496283] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3171.853318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.027293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3176.431879] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3181.242238] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lsmod [ 3183.479429] LustreError: 105054:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3183.483400] LustreError: 105055:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3183.493383] LustreError: 105054:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 3183.502108] LustreError: 105054:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3183.508790] LustreError: 105054:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3183.510932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3183.851722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3187.682505] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3187.692874] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.711189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.226518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3189.228131] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3189.238552] Lustre: Skipped 1 previous similar message [ 3189.260079] Lustre: Skipped 1 previous similar message [ 3192.311855] Lustre: server umount lustre-MDT0000 complete [ 3194.345378] LustreError: 103886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.381617] LustreError: 103886:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3195.803864] LustreError: 103868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156204 with bad export cookie 12946118304452248292 [ 3195.804475] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.812126] LustreError: 103868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3195.828385] Lustre: 103878:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3196.057658] Lustre: server umount lustre-MDT0001 complete [ 3208.926016] Lustre: server umount lustre-OST0000 complete [ 3215.664772] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3216.052692] LustreError: 106117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3216.132124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3219.713098] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3221.479130] LustreError: 106118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3225.571367] LustreError: 106117:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3226.891828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.222227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3231.038815] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3237.336248] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3242.358924] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3248.947060] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3249.302548] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3255.148219] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3260.608653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3262.726822] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3271.140199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3271.149863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.152245] Lustre: Skipped 1 previous similar message [ 3271.154501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3271.161596] Lustre: Skipped 1 previous similar message [ 3275.157489] Lustre: server umount lustre-OST0000 complete [ 3281.376768] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3281.382085] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3281.395228] Lustre: Skipped 1 previous similar message [ 3281.405112] Lustre: Skipped 1 previous similar message [ 3286.505882] LustreError: 106118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3286.533035] LustreError: 106118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3287.622550] Lustre: server umount lustre-MDT0000 complete [ 3291.510801] LustreError: 106099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156300 with bad export cookie 12946118304452248964 [ 3291.516525] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3291.523804] LustreError: 106099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3292.008536] Lustre: server umount lustre-MDT0001 complete [ 3302.928794] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 3305.413444] Key type lgssc unregistered [ 3305.782068] LNet: 109649:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3305.798632] LNetError: 109649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3305.837537] LNet: Removed LNI 192.168.203.107@tcp [ 3306.937702] Key type .llcrypt unregistered [ 3306.940418] Key type ._llcrypt unregistered [ 3332.699334] Key type ._llcrypt registered [ 3332.701252] Key type .llcrypt registered [ 3332.802583] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 3333.845914] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3333.907395] alg: No test for adler32 (adler32-zlib) [ 3334.983931] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3335.190164] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 3336.879556] Key type lgssc registered [ 3337.972084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3346.304916] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3346.331184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.720306] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.544723] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3358.183962] LustreError: 111070:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.216441] LustreError: 111070:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3361.948482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.394698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.804183] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3373.595262] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3379.863621] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3386.513624] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3386.794776] LustreError: 112887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3386.992640] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3393.603162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3393.658908] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3400.544342] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3406.379517] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3410.106741] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3413.985349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3413.992262] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3413.997811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3414.017821] Lustre: Skipped 1 previous similar message [ 3416.136818] Lustre: server umount lustre-OST0000 complete [ 3423.715893] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3423.731138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.743836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3423.748694] Lustre: Skipped 1 previous similar message [ 3426.621160] Lustre: server umount lustre-MDT0000 complete [ 3429.361794] LustreError: 111064:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.388210] LustreError: 111064:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3430.450586] LustreError: 111050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156439 with bad export cookie 11316263400694181991 [ 3430.453454] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3430.464360] LustreError: 111050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3430.744692] Lustre: server umount lustre-MDT0001 complete [ 3443.066482] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 3445.747392] Key type lgssc unregistered [ 3446.036646] LNet: 114706:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3446.046535] LNetError: 114706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3446.072699] LNet: Removed LNI 192.168.203.107@tcp [ 3446.942431] Key type .llcrypt unregistered [ 3446.944684] Key type ._llcrypt unregistered [ 3466.098678] Key type ._llcrypt registered [ 3466.104667] Key type .llcrypt registered [ 3466.241800] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 3467.622182] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3467.690841] alg: No test for adler32 (adler32-zlib) [ 3468.779789] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3469.010191] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 3470.689428] Key type lgssc registered [ 3472.037226] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3479.531686] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3479.552586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.932299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.577351] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3491.296413] LustreError: 115928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3491.309801] LustreError: 115928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3492.387511] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3492.704569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3497.058434] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3503.942175] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3509.816694] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3515.536182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3515.838937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3521.006804] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3527.091429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3527.451857] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3537.379700] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3537.381136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3537.406541] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3541.308411] Lustre: server umount lustre-OST0000 complete [ 3547.629573] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.648109] Lustre: Skipped 1 previous similar message [ 3547.653249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3547.657941] Lustre: Skipped 2 previous similar messages [ 3552.740885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3552.751133] Lustre: Skipped 1 previous similar message [ 3557.869663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3559.396172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3559.566525] Lustre: server umount lustre-MDT0000 complete [ 3562.977234] LustreError: 115927:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3562.992724] LustreError: 115927:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3563.556705] LustreError: 115909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156572 with bad export cookie 9007217813739000201 [ 3563.585423] LustreError: 115909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3563.585778] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3563.943142] Lustre: server umount lustre-MDT0001 complete [ 3576.233938] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 3579.746637] Key type lgssc unregistered [ 3580.097758] LNet: 119463:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3580.112765] LNetError: 119463:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3580.141969] LNet: Removed LNI 192.168.203.107@tcp [ 3581.082748] Key type .llcrypt unregistered [ 3581.085983] Key type ._llcrypt unregistered [ 3606.625647] Key type ._llcrypt registered [ 3606.631804] Key type .llcrypt registered [ 3606.832953] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 3608.031772] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3608.163500] alg: No test for adler32 (adler32-zlib) [ 3609.386918] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3609.680807] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 3611.343158] Key type lgssc registered [ 3612.386985] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3619.941971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3619.967712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3621.325874] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3625.029726] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3631.600718] LustreError: 120888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3631.622815] LustreError: 120888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3632.857381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3633.254514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3638.164181] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3645.355130] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3651.175866] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3658.053955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3658.212951] LustreError: 122711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3658.349549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3664.949129] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3666.934812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3672.530299] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3678.791329] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3682.647900] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3682.654166] Lustre: Skipped 2 previous similar messages [ 3687.175936] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3687.186230] Lustre: Skipped 2 previous similar messages [ 3688.010937] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3688.029369] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3700.571819] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.7@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3700.584585] Lustre: Skipped 2 previous similar messages [ 3707.956796] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.7@tcp 192.168.0.[2,10]@tcp [ 3713.632415] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.7/24@tcp in log lustre-MDT0000 [ 3713.636698] Lustre: Skipped 2 previous similar messages [ 3715.325231] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.7/24@tcp [ 3715.338850] Lustre: Skipped 1 previous similar message [ 3727.854212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3727.865078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3727.906537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3729.481263] Lustre: server umount lustre-OST0000 complete [ 3735.522781] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3735.527261] LustreError: Skipped 2 previous similar messages [ 3735.530935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3735.542864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.547467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3736.550986] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3739.607963] Lustre: server umount lustre-MDT0000 complete [ 3741.671677] LustreError: 120889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3741.690256] LustreError: 120889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3744.323559] LustreError: 120869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156753 with bad export cookie 9795509844052807819 [ 3744.329869] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3744.340700] LustreError: 120869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3755.171370] Lustre: server umount lustre-MDT0001 complete [ 3766.112055] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 3768.705717] Key type lgssc unregistered [ 3768.974369] LNet: 125729:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3768.979755] LNetError: 125729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3768.993337] LNet: Removed LNI 192.168.203.107@tcp [ 3769.770102] Key type .llcrypt unregistered [ 3769.772488] Key type ._llcrypt unregistered [ 3801.661572] Key type ._llcrypt registered [ 3801.664189] Key type .llcrypt registered [ 3801.787321] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 3802.560217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3802.587621] alg: No test for adler32 (adler32-zlib) [ 3803.765317] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3804.115785] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 3805.871409] Key type lgssc registered [ 3806.990678] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3814.707472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3814.734496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3816.160332] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3816.193213] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.7/24@tcp [ 3816.207796] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3821.091709] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3825.638578] LustreError: 127290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3825.670916] LustreError: 127290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3830.479620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3830.753891] LustreError: 127291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3831.150923] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3831.216211] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.7/24@tcp [ 3831.237371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3836.223523] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3845.096342] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3851.728245] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3858.742782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3859.239469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3861.301242] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 3866.056784] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 3874.763038] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3886.055625] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3886.061707] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3886.081518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3886.102359] Lustre: Skipped 1 previous similar message [ 3891.172180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3891.180504] Lustre: Skipped 1 previous similar message [ 3892.143332] Lustre: server umount lustre-OST0000 complete [ 3896.289058] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3896.295743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3896.303186] Lustre: Skipped 2 previous similar messages [ 3896.320970] Lustre: Skipped 1 previous similar message [ 3901.409074] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3902.362561] Lustre: server umount lustre-MDT0000 complete [ 3906.204801] LustreError: 127270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156914 with bad export cookie 1654680220059180106 [ 3906.219273] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3906.221870] LustreError: 127270:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3906.620108] Lustre: server umount lustre-MDT0001 complete [ 3918.483667] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 3921.030723] Key type lgssc unregistered [ 3921.387325] LNet: 130881:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3921.421124] LNetError: 130881:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3921.438775] LNet: Removed LNI 192.168.203.107@tcp [ 3922.315656] Key type .llcrypt unregistered [ 3922.317887] Key type ._llcrypt unregistered [ 3947.928427] Key type ._llcrypt registered [ 3947.931468] Key type .llcrypt registered [ 3948.021858] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 3964.629618] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 3966.511296] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3966.538985] alg: No test for adler32 (adler32-zlib) [ 3967.852849] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3968.156609] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 3969.951586] Key type lgssc registered [ 3971.089297] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4015.392787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4015.428756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4016.681273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4016.700812] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4016.751950] Lustre: lustre-MDT0000: new disk, initializing [ 4016.806316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4016.850601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4022.442654] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4033.129034] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4033.252983] Lustre: 134290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4033.296515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4033.303260] Lustre: Skipped 1 previous similar message [ 4033.373565] Lustre: lustre-MDT0001: new disk, initializing [ 4033.462462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4033.495978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4033.503214] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4037.645121] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4046.403923] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4052.301080] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4058.984192] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4059.196729] Lustre: lustre-OST0000: new disk, initializing [ 4059.201866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4059.208752] Lustre: 136258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4059.270805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.890519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4064.906516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4065.060253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4065.812303] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4079.097826] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4084.297429] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4084.613758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4090.026807] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4090.187341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4095.977423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.988238] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4096.012155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4096.023580] LustreError: Skipped 1 previous similar message [ 4101.088933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4101.101555] Lustre: Skipped 2 previous similar messages [ 4106.207332] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4106.212695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4106.230320] Lustre: Skipped 1 previous similar message [ 4106.362784] Lustre: server umount lustre-OST0000 complete [ 4111.330366] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4111.338209] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4111.366276] Lustre: Skipped 2 previous similar messages [ 4111.391994] Lustre: Skipped 1 previous similar message [ 4116.449423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4116.458966] Lustre: Skipped 1 previous similar message [ 4117.311564] Lustre: server umount lustre-MDT0000 complete [ 4121.452326] LustreError: 135085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157130 with bad export cookie 9352128553764581636 [ 4121.464573] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4121.467896] LustreError: 135085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4121.845448] Lustre: server umount lustre-MDT0001 complete [ 4132.014738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4132.650654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4137.709516] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4137.953908] LustreError: 138474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4137.973170] LustreError: 138474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4143.078156] LustreError: 138475:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4147.629416] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4148.312669] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4155.251946] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4164.893605] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4171.562103] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4178.928267] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4179.459047] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4186.756681] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4195.445134] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4200.616892] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4204.104261] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4209.552338] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4209.863955] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4220.300668] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4220.502953] Lustre: lustre-OST0001: new disk, initializing [ 4220.507384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4220.516484] Lustre: 141868:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4220.645113] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4228.314686] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4230.264469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4230.276896] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4230.380323] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4238.422531] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4238.740049] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4244.710730] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4245.165557] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4251.593653] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff918eccadd000.ost_server_uuid 50 [ 4253.262944] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff918eccadd000.ost_server_uuid in IDLE state after 0 sec [ 4266.506248] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4266.508935] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4266.519481] LustreError: Skipped 1 previous similar message [ 4266.520360] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4266.537655] Lustre: Skipped 1 previous similar message [ 4268.882403] Lustre: server umount lustre-OST0001 complete [ 4271.601285] LustreError: 141217:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4271.637715] LustreError: 141217:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4274.657078] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4274.662778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.734304] LustreError: 140297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4276.749575] LustreError: 140297:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4278.963366] Lustre: server umount lustre-OST0000 complete [ 4286.432714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4286.440300] LustreError: Skipped 1 previous similar message [ 4286.442602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4286.468381] Lustre: Skipped 1 previous similar message [ 4286.481262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4286.491089] Lustre: Skipped 5 previous similar messages [ 4297.183139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4297.559100] Lustre: server umount lustre-MDT0000 complete [ 4301.781919] LustreError: 141867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157310 with bad export cookie 9352128553764582518 [ 4301.790026] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4301.801649] LustreError: 141867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4302.175627] Lustre: server umount lustre-MDT0001 complete [ 4326.297257] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 4329.747456] Key type lgssc unregistered [ 4330.021562] LNet: 144877:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4330.030393] LNetError: 144877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4330.049951] LNet: Removed LNI 192.168.203.107@tcp [ 4331.376947] Key type .llcrypt unregistered [ 4331.378428] Key type ._llcrypt unregistered [ 4378.680451] Key type ._llcrypt registered [ 4378.683730] Key type .llcrypt registered [ 4378.834483] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 4397.095732] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 4399.585492] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4399.635251] alg: No test for adler32 (adler32-zlib) [ 4400.883565] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4401.251417] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 4402.944273] Key type lgssc registered [ 4404.062659] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4442.770270] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4442.827106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.183852] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4444.270287] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4444.388905] Lustre: lustre-MDT0000: new disk, initializing [ 4444.512861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4444.522830] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4448.959846] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4460.331151] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4460.516258] Lustre: 149034:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4460.578887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4460.584275] Lustre: Skipped 1 previous similar message [ 4460.693936] Lustre: lustre-MDT0001: new disk, initializing [ 4460.805191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4460.852808] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4460.865242] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4465.309702] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4475.794316] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4481.478685] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4488.252675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.518947] Lustre: lustre-OST0000: new disk, initializing [ 4488.522325] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4488.536774] Lustre: 151003:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4488.699294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4497.426437] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4497.543974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4497.564809] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4497.801140] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4507.747464] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4513.316845] Lustre: Failing over lustre-OST0000 [ 4513.468599] Lustre: server umount lustre-OST0000 complete [ 4530.656377] Lustre: 147258:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157523/real 1788157523] req@0000000009bdf6b2 x1875018943334144/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788157539 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4530.726043] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.157928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4534.755684] Lustre: 147260:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157527/real 1788157527] req@00000000166cfbf3 x1875018943335424/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788157543 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4534.879303] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4535.006723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4535.042413] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4536.033955] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4539.871139] Lustre: 147260:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157532/real 1788157532] req@00000000369a5128 x1875018943336576/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788157548 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4539.914564] Lustre: 147260:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4542.467223] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4542.470875] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4542.613253] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4544.543163] Lustre: 147261:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157537/real 1788157537] req@00000000d8cbfe1b x1875018943337600/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788157553 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4544.579691] Lustre: 147261:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4547.095330] Lustre: Failing over lustre-MDT0000 [ 4547.660119] Lustre: server umount lustre-MDT0000 complete [ 4548.066036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4548.079299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.654554] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4551.664354] Lustre: Skipped 1 previous similar message [ 4552.642337] LustreError: 151181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4552.664223] LustreError: 151181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 4556.773434] LustreError: 149041:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4556.797554] LustreError: 149041:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4561.902521] LustreError: 149047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4561.941484] LustreError: 149047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4567.016147] LustreError: 150405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4567.068490] LustreError: 150405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4568.034399] Lustre: 147260:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788157560/real 1788157560] req@000000005eeca949 x1875018943348992/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788157576 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4568.058614] Lustre: 147260:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4568.066097] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4568.983763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4573.122102] LustreError: 151181:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4573.170424] LustreError: 151181:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4578.736810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4578.820681] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4582.877576] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4583.927703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4583.942559] Lustre: Skipped 1 previous similar message [ 4583.971658] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4584.019775] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4584.834964] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4594.656234] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4594.666499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4594.696793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4598.754167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4598.778881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4600.491799] Lustre: server umount lustre-OST0000 complete [ 4609.506205] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4609.509820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4609.521036] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4609.544988] Lustre: Skipped 2 previous similar messages [ 4609.548462] Lustre: Skipped 2 previous similar messages [ 4611.043426] Lustre: server umount lustre-MDT0000 complete [ 4614.627855] LustreError: 150243:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4614.655256] LustreError: 150243:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 4615.510770] LustreError: 149026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157624 with bad export cookie 10349144307917860099 [ 4615.513498] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4615.527107] LustreError: 149026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4615.858519] Lustre: server umount lustre-MDT0001 complete [ 4626.631908] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 4629.349468] Key type lgssc unregistered [ 4629.820081] LNet: 154660:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4629.834309] LNetError: 154660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4630.885907] LNet: Removed LNI 192.168.203.107@tcp [ 4631.867413] Key type .llcrypt unregistered [ 4631.870585] Key type ._llcrypt unregistered [ 4672.497782] Key type ._llcrypt registered [ 4672.501559] Key type .llcrypt registered [ 4672.600700] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 4674.171113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4674.251775] alg: No test for adler32 (adler32-zlib) [ 4675.419631] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4675.687949] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 4677.383151] Key type lgssc registered [ 4678.549705] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4687.326634] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4687.363758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4689.085201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4693.871212] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4698.595997] LustreError: 156186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4698.614467] LustreError: 156186:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4703.722325] LustreError: 156187:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4704.682657] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4708.837176] LustreError: 156186:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4709.147661] LustreError: 156215:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4709.165068] LustreError: 156215:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4709.173204] LustreError: 156215:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4709.404987] Lustre: server umount lustre-MDT0000 complete [ 4726.016547] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 4736.742453] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 4777.524262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4777.765035] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4777.811567] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4777.904039] Lustre: lustre-MDT0000: new disk, initializing [ 4778.008673] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4778.035598] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4782.760805] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4794.491489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4794.631380] Lustre: 159516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4794.683232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4794.688465] Lustre: Skipped 1 previous similar message [ 4794.795633] Lustre: lustre-MDT0001: new disk, initializing [ 4794.920144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4794.966642] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4795.002631] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4799.647845] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4811.426405] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4819.824790] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4826.846663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4827.059948] Lustre: lustre-OST0000: new disk, initializing [ 4827.063202] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4827.068664] Lustre: 161487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4827.152469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4833.864564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4833.886948] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4834.025841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4834.319889] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 4845.575910] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4850.760407] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4852.989521] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:31:00 (1788157860) [ 5126.111465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5126.111743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5126.119391] Lustre: Skipped 2 previous similar messages [ 5126.125453] Lustre: Skipped 2 previous similar messages [ 5130.614685] Lustre: server umount lustre-MDT0000 complete [ 5131.234060] LustreError: 161044:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5131.261767] LustreError: 161044:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5133.958280] LustreError: 159511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158142 with bad export cookie 6902323172182367425 [ 5133.962546] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5133.969336] LustreError: 159511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5136.351727] LustreError: 163261:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5136.352784] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.369781] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5138.400059] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5140.320201] Lustre: server umount lustre-MDT0001 complete [ 5158.368261] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5158.537532] Lustre: server umount lustre-OST0000 complete [ 5167.401433] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 5177.259267] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 5210.431096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5210.803394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5210.822527] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5210.874429] Lustre: lustre-MDT0000: new disk, initializing [ 5210.952793] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5210.966457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5216.016413] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5228.435417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.655076] Lustre: 166156:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5228.719658] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5228.727155] Lustre: Skipped 1 previous similar message [ 5228.844295] Lustre: lustre-MDT0001: new disk, initializing [ 5228.990926] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5229.037927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5229.051899] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5233.178198] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5241.917590] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5248.311694] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5255.108383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5255.423602] Lustre: lustre-OST0000: new disk, initializing [ 5255.426538] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5255.433817] Lustre: 168125:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5255.514637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5261.415431] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5263.435108] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5263.444398] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5263.528129] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5270.385627] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5275.012643] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5275.198557] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5279.198637] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5279.340811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5284.320179] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5284.325595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5284.327739] LustreError: Skipped 1 previous similar message [ 5284.341290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5284.351519] Lustre: Skipped 1 previous similar message [ 5287.460911] Lustre: server umount lustre-OST0000 complete [ 5294.563066] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5294.570100] Lustre: Skipped 1 previous similar message [ 5294.579616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5294.591022] Lustre: Skipped 1 previous similar message [ 5304.799901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5304.805626] Lustre: Skipped 3 previous similar messages [ 5305.311131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5305.495460] Lustre: server umount lustre-MDT0000 complete [ 5309.375886] LustreError: 166148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158318 with bad export cookie 6902323172182526255 [ 5309.381091] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5309.390624] LustreError: 166148:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5309.647253] Lustre: server umount lustre-MDT0001 complete [ 5317.856890] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:38:45 (1788158325) [ 5332.029516] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 5340.199683] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 5374.726179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5374.993572] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5375.034469] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5375.165747] Lustre: lustre-MDT0000: new disk, initializing [ 5375.267979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5375.290140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5379.542246] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5389.782534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5389.910642] Lustre: 172880:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5390.131197] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5394.436452] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5404.021677] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5409.714763] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5416.870850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5417.279269] Lustre: lustre-OST0000: new disk, initializing [ 5417.286271] Lustre: Skipped 1 previous similar message [ 5417.295461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5417.299792] Lustre: Skipped 2 previous similar messages [ 5417.305480] Lustre: 174858:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5423.910347] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5427.767756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5427.779031] Lustre: Skipped 1 previous similar message [ 5427.788256] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5427.876367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5433.133337] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5443.555353] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5443.563125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5443.571689] Lustre: Skipped 1 previous similar message [ 5443.577279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5446.331386] Lustre: server umount lustre-OST0000 complete [ 5451.744733] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5451.754140] LustreError: Skipped 1 previous similar message [ 5451.757872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5451.769696] Lustre: Skipped 1 previous similar message [ 5456.367548] Lustre: server umount lustre-MDT0000 complete [ 5458.923200] LustreError: 173696:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5458.955979] LustreError: 173696:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5459.955076] LustreError: 172874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158468 with bad export cookie 6902323172182527137 [ 5459.959321] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5459.969575] LustreError: 172874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5460.248558] Lustre: server umount lustre-MDT0001 complete [ 5468.683588] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:41:16 (1788158476) [ 5483.758159] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 5494.968630] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 5534.131253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5534.298732] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5534.320489] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5534.367672] Lustre: lustre-MDT0000: new disk, initializing [ 5534.410164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5534.414380] Lustre: Skipped 2 previous similar messages [ 5534.423790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5537.955502] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5548.821748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5549.011613] Lustre: 179179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5549.411751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5554.603346] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5565.205959] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5571.876743] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5578.901721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5579.112918] Lustre: 181159:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5584.790711] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5589.072235] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5589.183454] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5593.798892] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5604.848075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5604.848946] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.856843] LustreError: Skipped 1 previous similar message [ 5604.857727] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5604.874347] Lustre: Skipped 2 previous similar messages [ 5604.900945] Lustre: Skipped 5 previous similar messages [ 5607.031206] Lustre: server umount lustre-OST0000 complete [ 5610.981574] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5616.860258] Lustre: server umount lustre-MDT0000 complete [ 5620.062165] LustreError: 179981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158628 with bad export cookie 6902323172182528222 [ 5620.062760] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5620.078592] LustreError: 179981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5620.409446] Lustre: server umount lustre-MDT0001 complete [ 5629.799570] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 5632.269333] Key type lgssc unregistered [ 5632.518227] LNet: 183016:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5632.529674] LNetError: 183016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5632.545475] LNet: Removed LNI 192.168.203.107@tcp [ 5633.326438] Key type .llcrypt unregistered [ 5633.329467] Key type ._llcrypt unregistered [ 5655.725224] Key type ._llcrypt registered [ 5655.727641] Key type .llcrypt registered [ 5655.888504] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 5656.902017] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5657.023325] alg: No test for adler32 (adler32-zlib) [ 5658.071057] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5658.278046] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 5659.943270] Key type lgssc registered [ 5661.063482] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5667.973791] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5667.998709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5669.596093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5673.744597] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5680.106554] LustreError: 184437:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5680.148433] LustreError: 184437:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5682.024643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.376751] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5686.613720] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5693.571975] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5699.346562] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5705.102811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5705.468611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5710.621340] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5714.670239] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5718.211182] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5737.440642] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5737.456169] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5737.470736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5740.515392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5740.521386] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5740.530523] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5742.206982] Lustre: server umount lustre-OST0000 complete [ 5745.635289] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.639411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5745.648893] Lustre: Skipped 1 previous similar message [ 5745.659175] Lustre: Skipped 1 previous similar message [ 5750.756851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5750.769741] Lustre: Skipped 2 previous similar messages [ 5751.897265] Lustre: server umount lustre-MDT0000 complete [ 5755.474216] LustreError: 184418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158764 with bad export cookie 13262134166997373060 [ 5755.487717] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5755.490669] LustreError: 184418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5755.829964] Lustre: server umount lustre-MDT0001 complete [ 5766.137209] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 5768.721693] Key type lgssc unregistered [ 5769.002251] LNet: 187985:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5769.012250] LNetError: 187985:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5769.024595] LNet: Removed LNI 192.168.203.107@tcp [ 5769.805399] Key type .llcrypt unregistered [ 5769.808986] Key type ._llcrypt unregistered [ 5792.850797] Key type ._llcrypt registered [ 5792.852570] Key type .llcrypt registered [ 5792.957380] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 5794.041444] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5794.213786] alg: No test for adler32 (adler32-zlib) [ 5795.314478] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 5795.633373] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 5797.335163] Key type lgssc registered [ 5798.522878] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5806.821846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5806.839605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5808.404708] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5813.298636] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5818.859825] LustreError: 189406:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5818.874585] LustreError: 189406:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5822.063510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.494084] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5826.934823] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5835.073169] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5840.171069] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5846.656505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5847.021990] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5853.315759] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5854.215597] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5860.069538] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5866.967826] Lustre: lustre-OST0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 5868.515325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5868.535195] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5868.549176] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5869.537817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5869.558063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5869.572700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5869.587481] Lustre: Skipped 1 previous similar message [ 5871.747746] Lustre: server umount lustre-OST0000 complete [ 5878.134829] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff918ec560c800.ost_server_uuid 50 [ 5879.477499] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5894.113273] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5894.120221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5894.130546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5899.553189] Lustre: server umount lustre-MDT0000 complete [ 5900.264486] LustreError: 189403:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.286671] LustreError: 189403:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5903.076632] LustreError: 189386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158911 with bad export cookie 12691561536592470585 [ 5903.089204] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5903.338987] Lustre: server umount lustre-MDT0001 complete [ 5911.946051] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:48:40 (1788158920) [ 5917.979300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5918.311426] LustreError: 193319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.372806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5922.118971] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5923.814913] LustreError: 193320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.930367] LustreError: 193319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5931.487689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5932.166374] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5936.770551] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5944.615340] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5950.549265] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5956.470579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5956.767649] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5962.338948] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5963.963416] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5969.507536] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5976.547004] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5976.778959] Lustre: lustre-OST0001: new disk, initializing [ 5976.788762] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5976.800516] Lustre: 196161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5976.882379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5983.657454] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 5983.842297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5983.857988] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5983.998445] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5992.933491] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5998.555439] Lustre: lustre-OST0000: Not available for connect from 192.168.203.7@tcp (stopping) [ 5998.568359] Lustre: Skipped 2 previous similar messages [ 5999.080606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5999.089945] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5999.104868] Lustre: Skipped 1 previous similar message [ 6001.574195] Lustre: server umount lustre-OST0000 complete [ 6003.656547] LustreError: 195139:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6010.131396] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6010.359953] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6017.027966] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6017.312477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6020.070948] LustreError: 196445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6020.092657] LustreError: 196445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6035.433075] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6035.438324] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6035.444199] LustreError: Skipped 1 previous similar message [ 6035.445451] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6035.445457] Lustre: Skipped 2 previous similar messages [ 6035.473132] Lustre: Skipped 2 previous similar messages [ 6037.544349] Lustre: server umount lustre-OST0001 complete [ 6044.640641] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6044.656309] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6047.886062] Lustre: server umount lustre-MDT0000 complete [ 6051.095834] LustreError: 195163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159059 with bad export cookie 12691561536592471656 [ 6051.096659] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6051.107855] LustreError: 195163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 6051.331967] Lustre: server umount lustre-MDT0001 complete [ 6072.025571] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:51:20 (1788159080) [ 6079.621184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6079.837103] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6079.888493] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6080.075151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6084.044307] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6094.077709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.190559] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6094.243456] Lustre: 199603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6099.340679] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6107.232392] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6113.745511] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6120.530425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6120.720723] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6126.823550] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6130.875965] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6134.137367] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6141.082416] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6141.161634] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6147.733972] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6155.448152] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6160.881669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6160.889595] Lustre: Skipped 1 previous similar message [ 6160.894815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6160.904765] Lustre: Skipped 5 previous similar messages [ 6164.345114] Lustre: server umount lustre-OST0000 complete [ 6166.012069] LustreError: 201456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6166.034989] LustreError: 201456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6185.955449] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6188.552679] Lustre: server umount lustre-OST0001 complete [ 6196.707368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.710707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6196.743065] Lustre: Skipped 4 previous similar messages [ 6196.764251] Lustre: Skipped 5 previous similar messages [ 6198.775350] Lustre: server umount lustre-MDT0000 complete [ 6203.120786] LustreError: 203674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159211 with bad export cookie 12691561536592473063 [ 6203.126103] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6203.130112] LustreError: 203674:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6203.377384] Lustre: server umount lustre-MDT0001 complete [ 6228.432450] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:53:56 (1788159236) [ 6244.105891] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 6253.600923] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 6285.908130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6286.188622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6286.210566] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6286.272358] Lustre: lustre-MDT0000: new disk, initializing [ 6286.315189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6286.318980] Lustre: Skipped 3 previous similar messages [ 6286.330961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6291.077435] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6301.235436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6301.409573] Lustre: 207703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6301.482511] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6301.486336] Lustre: Skipped 1 previous similar message [ 6301.547359] Lustre: lustre-MDT0001: new disk, initializing [ 6301.705861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6301.722283] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6306.349374] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6315.850519] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6321.711982] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6327.449393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6327.751264] Lustre: lustre-OST0000: new disk, initializing [ 6327.754819] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6327.760642] Lustre: 209668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6334.557939] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6337.081479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6337.097252] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6337.211297] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6344.454919] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6349.814181] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6350.110716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6354.934295] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6355.173650] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6357.986034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6358.006270] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6358.007700] Lustre: Skipped 1 previous similar message [ 6358.013413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6358.028579] LustreError: Skipped 2 previous similar messages [ 6358.040685] Lustre: Skipped 1 previous similar message [ 6371.295884] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6371.564213] Lustre: server umount lustre-OST0000 complete [ 6381.648105] Lustre: server umount lustre-MDT0000 complete [ 6383.594892] LustreError: 207713:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.630333] LustreError: 207713:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 14 previous similar messages [ 6385.218205] LustreError: 207693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159393 with bad export cookie 12691561536592474624 [ 6385.226501] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6385.231278] LustreError: 207693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6385.559847] Lustre: server umount lustre-MDT0001 complete [ 6393.808591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6399.261392] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6407.592022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.724453] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6421.595567] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6427.643804] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6433.798924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6434.156344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6434.166892] Lustre: Skipped 4 previous similar messages [ 6440.059615] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6448.185804] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6453.671532] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6453.896554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6458.827021] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6459.030089] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6462.953623] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6462.964953] LustreError: Skipped 1 previous similar message [ 6467.444124] Lustre: server umount lustre-OST0000 complete [ 6473.849586] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6478.539391] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6507.631593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.815318] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6522.052106] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6525.970882] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6527.477918] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6527.503265] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6527.503305] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6532.577587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6532.589804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6532.609271] Lustre: Skipped 5 previous similar messages [ 6532.635979] Lustre: Skipped 8 previous similar messages [ 6536.632637] Lustre: server umount lustre-OST0000 complete [ 6546.915205] Lustre: server umount lustre-MDT0000 complete [ 6547.943310] LustreError: 211889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.953336] LustreError: 211889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 6550.533159] LustreError: 211869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159559 with bad export cookie 12691561536592475506 [ 6550.538283] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6550.548724] LustreError: 211869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6550.990713] Lustre: server umount lustre-MDT0001 complete [ 6560.015717] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:59:27 (1788159567) [ 6566.568645] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.473453] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6579.334563] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6583.945554] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6590.456727] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6595.255578] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6601.212049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6606.591547] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6613.629429] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6617.719723] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6617.904076] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6621.523424] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6621.678960] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6627.327690] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6627.505988] Lustre: lustre-OST0001: new disk, initializing [ 6627.509446] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6627.513971] Lustre: 221223:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6632.714764] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6633.031283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6633.044393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6633.124231] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6640.878864] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6644.562911] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6644.792918] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6649.536896] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6649.760751] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6653.926706] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6653.945763] LustreError: Skipped 2 previous similar messages [ 6658.512193] Lustre: server umount lustre-OST0001 complete [ 6666.135221] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6666.377639] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6671.064261] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6671.436355] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6700.143544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6700.278594] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6700.286223] Lustre: Skipped 5 previous similar messages [ 6705.514668] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6705.641941] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6705.648707] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6705.652732] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6705.661914] Lustre: Skipped 2 previous similar messages [ 6712.341453] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6714.107688] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6730.207911] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6730.402303] Lustre: server umount lustre-OST0001 complete [ 6753.582317] Lustre: server umount lustre-OST0000 complete [ 6765.052514] Lustre: server umount lustre-MDT0000 complete [ 6768.537726] LustreError: 217837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159777 with bad export cookie 12691561536592476472 [ 6768.548447] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6768.549068] LustreError: 217837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6774.922339] Lustre: server umount lustre-MDT0001 complete [ 6795.806909] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:03:23 (1788159803) [ 6801.653314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6801.906593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6801.929110] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6806.031153] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6814.062162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6814.200498] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6814.233129] Lustre: 226482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6818.681659] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6825.776378] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6831.038962] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6837.275949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6837.449255] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6843.626986] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6850.877060] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6860.187980] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6860.266534] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6866.544799] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6874.170034] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6875.578075] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6879.778276] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6879.987843] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6884.607588] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6884.796774] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6890.494966] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff918ecc27d800.ost_server_uuid 50 [ 6892.107335] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff918ecc27d800.ost_server_uuid in IDLE state after 0 sec [ 6893.066559] Lustre: Permanently deactivating lustre-OST0001 [ 6893.095672] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6897.003370] Lustre: Permanently reactivating lustre-OST0001 [ 6901.223736] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6901.230143] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6901.262099] Lustre: Skipped 20 previous similar messages [ 6901.262909] Lustre: Skipped 11 previous similar messages [ 6902.922850] Lustre: server umount lustre-OST0001 complete [ 6906.347737] LustreError: 229855:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6906.366606] LustreError: 229855:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 41 previous similar messages [ 6912.440413] Lustre: server umount lustre-OST0000 complete [ 6930.399134] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6930.689180] Lustre: server umount lustre-MDT0000 complete [ 6933.566093] LustreError: 226473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159942 with bad export cookie 12691561536592477760 [ 6933.569958] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6933.581534] LustreError: 226473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6933.779825] Lustre: server umount lustre-MDT0001 complete [ 6953.205386] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:06:01 (1788159961) [ 6959.179163] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6959.362326] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6959.408912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6959.416704] Lustre: Skipped 5 previous similar messages [ 6965.320688] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6973.732051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6974.014826] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6974.045763] Lustre: 233170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6978.158840] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 6985.250337] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6990.831050] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6997.394985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6997.605209] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6997.629881] Lustre: Permanently deactivating lustre-OST0000 [ 6997.641594] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7003.377161] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7006.762246] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7006.854918] LustreError: 235564:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 7010.506319] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7017.012394] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7017.068147] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7023.187824] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7023.319213] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7030.658758] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7032.410805] Lustre: Permanently reactivating lustre-OST0000 [ 7032.430427] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7032.442489] Lustre: Skipped 2 previous similar messages [ 7038.520683] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7038.531760] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp 000000002bc5e728, cur 1788160047 deadline 1788160045 last 1788160015 [ 7038.588387] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7048.238274] Lustre: server umount lustre-OST0001 complete [ 7049.186471] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7049.194674] LustreError: Skipped 6 previous similar messages [ 7057.790026] Lustre: server umount lustre-OST0000 complete [ 7076.319127] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7076.583699] Lustre: server umount lustre-MDT0000 complete [ 7080.505737] LustreError: 233163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160089 with bad export cookie 12691561536592479433 [ 7080.507288] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7080.521684] LustreError: 233163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7080.858602] Lustre: server umount lustre-MDT0001 complete [ 7092.097874] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 7094.684567] Key type lgssc unregistered [ 7094.980995] LNet: 238051:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7094.989857] LNetError: 238051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7095.014903] LNet: Removed LNI 192.168.203.107@tcp [ 7095.865106] Key type .llcrypt unregistered [ 7095.866669] Key type ._llcrypt unregistered [ 7116.784219] Key type ._llcrypt registered [ 7116.789958] Key type .llcrypt registered [ 7116.896847] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 7117.916753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7117.957789] alg: No test for adler32 (adler32-zlib) [ 7118.943689] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7119.188723] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 7120.847177] Key type lgssc registered [ 7121.899124] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7130.932711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7130.953635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7132.330751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7135.720503] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7141.862690] LustreError: 239522:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7141.891210] LustreError: 239522:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7142.887839] LustreError: 239521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7143.702658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7143.851626] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7143.878807] Lustre: 239510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7143.889643] Lustre: Permanently deactivating lustre-MDT0001 [ 7143.905470] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7144.328458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7148.551214] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7148.561299] LustreError: 239552:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7148.584052] LustreError: 239552:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7148.603677] LustreError: 239552:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 7148.912665] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7156.334138] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7162.509895] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7170.557615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7171.253865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7176.397759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7177.399131] LustreError: 241367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7179.060416] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7182.827110] LustreError: 241366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7187.841433] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7187.946948] LustreError: 241364:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7188.993812] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:193) [ 7194.278984] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.506903] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7199.380853] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7199.725573] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7199.738455] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7205.289302] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7209.412337] Lustre: Permanently reactivating lustre-MDT0001 [ 7209.455059] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7209.467882] Lustre: Skipped 2 previous similar messages [ 7215.057641] Lustre: lustre-MDT0001: Client deea59b0-cc48-422a-a525-909d08642833 (at 192.168.203.7@tcp) reconnecting [ 7219.586828] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7220.867267] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7225.724567] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7225.924286] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7230.245598] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7230.403386] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7233.037691] Lustre: Permanently deactivating lustre-MDT0001 [ 7233.053987] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7233.060256] Lustre: Skipped 2 previous similar messages [ 7242.346591] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7250.916589] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7250.919468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7250.931890] Lustre: Skipped 2 previous similar messages [ 7250.951046] Lustre: Skipped 3 previous similar messages [ 7254.215884] Lustre: server umount lustre-MDT0000 complete [ 7256.034393] LustreError: 239521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7256.048772] LustreError: 239521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 7257.451405] LustreError: 240261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160266 with bad export cookie 14579598723049960 [ 7257.455536] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7257.462735] LustreError: 240261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7257.713540] Lustre: server umount lustre-MDT0001 complete [ 7263.719803] Lustre: server umount lustre-OST0000 complete [ 7273.851856] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:11:21 (1788160281) [ 7283.087395] Lustre: server umount lustre-OST0001 complete [ 7289.522246] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 7299.029725] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 7331.591251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.766798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7331.776491] Lustre: Skipped 2 previous similar messages [ 7331.799781] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7331.865890] Lustre: lustre-MDT0000: new disk, initializing [ 7331.929439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7331.941514] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7335.796665] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7345.307399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.445968] Lustre: 247787:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7345.503508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7345.508510] Lustre: Skipped 1 previous similar message [ 7345.615687] Lustre: lustre-MDT0001: new disk, initializing [ 7345.727252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7345.770697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7345.784216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7349.564990] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7358.644753] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7363.785066] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7369.834843] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7370.030105] Lustre: lustre-OST0000: new disk, initializing [ 7370.035747] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7370.046536] Lustre: 249757:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7370.133923] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7375.582661] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7379.026082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7379.039204] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7379.148053] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7384.812385] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7387.312299] LustreError: 248974:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7395.558792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7395.776918] Lustre: lustre-OST0001: new disk, initializing [ 7395.779668] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7395.785484] Lustre: 250873:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7397.351416] LustreError: 248974:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7397.404877] LustreError: 247795:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7401.894986] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7403.614250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7403.630659] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7403.787492] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7407.488691] LustreError: 247795:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7410.779818] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7412.405135] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7414.236271] Lustre: lustre-OST0001: Not available for connect from 192.168.203.7@tcp (stopping) [ 7414.244467] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7414.245188] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7414.253019] LustreError: Skipped 1 previous similar message [ 7414.259215] Lustre: Skipped 2 previous similar messages [ 7419.327133] Lustre: lustre-OST0001: Not available for connect from 192.168.203.7@tcp (stopping) [ 7419.334689] Lustre: Skipped 2 previous similar messages [ 7420.347923] Lustre: server umount lustre-OST0001 complete [ 7424.444440] LustreError: 249742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7439.858080] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7439.861530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7439.867024] Lustre: Skipped 1 previous similar message [ 7439.878907] Lustre: Skipped 3 previous similar messages [ 7443.887773] Lustre: server umount lustre-OST0000 complete [ 7448.034210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7448.044930] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7448.062491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7453.462495] LustreError: 252160:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (000000008aab91cf) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7453.532986] Lustre: server umount lustre-MDT0000 complete [ 7457.361302] LustreError: 248581:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160466 with bad export cookie 14579598723052998 [ 7457.369244] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7463.839483] Lustre: server umount lustre-MDT0001 complete [ 7474.722880] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 7477.091367] Key type lgssc unregistered [ 7477.349472] LNet: 252917:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7477.358201] LNetError: 252917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7477.371301] LNet: Removed LNI 192.168.203.107@tcp [ 7478.270160] Key type .llcrypt unregistered [ 7478.274669] Key type ._llcrypt unregistered [ 7511.578477] Key type ._llcrypt registered [ 7511.580351] Key type .llcrypt registered [ 7511.674448] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 7512.550765] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7512.580681] alg: No test for adler32 (adler32-zlib) [ 7513.598937] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7513.851541] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 7515.543177] Key type lgssc registered [ 7516.525968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7523.853481] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7523.869283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7525.038634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7525.065708] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7525.190743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7528.980577] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7537.289272] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7537.459845] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7537.495225] Lustre: 255103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7537.769190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7543.076915] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7549.902975] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7555.516205] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7562.677861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7562.966634] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7563.219427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7570.928633] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7571.618679] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7580.078771] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7628.268145] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7628.271633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7628.300333] Lustre: Skipped 1 previous similar message [ 7628.301017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7628.323498] Lustre: Skipped 1 previous similar message [ 7633.389775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7633.397159] Lustre: Skipped 1 previous similar message [ 7634.046243] Lustre: server umount lustre-OST0000 complete [ 7638.142715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7649.660236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7650.042858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7656.017333] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7659.248750] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7659.264898] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7659.274785] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7659.299204] Lustre: Skipped 1 previous similar message [ 7663.861866] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7705.575451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7705.579958] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7705.592232] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7705.602119] Lustre: Skipped 1 previous similar message [ 7705.617996] Lustre: Skipped 1 previous similar message [ 7709.695577] Lustre: server umount lustre-OST0000 complete [ 7715.812616] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7715.833357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7719.442347] Lustre: server umount lustre-MDT0000 complete [ 7722.800313] LustreError: 256963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160731 with bad export cookie 6087341819372770421 [ 7722.805388] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7722.826268] LustreError: 256963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7723.105174] Lustre: server umount lustre-MDT0001 complete [ 7734.014694] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 7736.593979] Key type lgssc unregistered [ 7736.822567] LNet: 260523:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7736.832441] LNetError: 260523:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7736.844691] LNet: Removed LNI 192.168.203.107@tcp [ 7737.581173] Key type .llcrypt unregistered [ 7737.587027] Key type ._llcrypt unregistered [ 7766.188472] Key type ._llcrypt registered [ 7766.190787] Key type .llcrypt registered [ 7766.325134] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 7767.452091] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7767.485988] alg: No test for adler32 (adler32-zlib) [ 7768.473604] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7768.738466] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 7770.472145] Key type lgssc registered [ 7771.571690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7779.089530] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7779.108494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7780.497205] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7784.351134] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7791.090351] LustreError: 261941:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7791.117670] LustreError: 261941:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7792.938863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.222579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7797.220320] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7805.235082] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7811.243511] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7818.321035] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7818.775971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7825.973739] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7826.336731] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7834.655942] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7861.741187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7861.755937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7861.773777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7863.777121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7863.780517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7863.784279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7863.806034] Lustre: Skipped 1 previous similar message [ 7867.296947] Lustre: server umount lustre-OST0000 complete [ 7874.017076] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.019870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7874.045515] Lustre: Skipped 1 previous similar message [ 7874.083603] Lustre: Skipped 1 previous similar message [ 7877.294497] Lustre: server umount lustre-MDT0000 complete [ 7879.138619] LustreError: 261938:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7879.161546] LustreError: 261938:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7880.909989] LustreError: 262677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160889 with bad export cookie 15018135608816507584 [ 7880.918036] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7880.926327] LustreError: 262677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7881.254850] Lustre: server umount lustre-MDT0001 complete [ 7891.497562] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 7894.062978] Key type lgssc unregistered [ 7894.365558] LNet: 266248:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7894.370714] LNetError: 266248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7894.385611] LNet: Removed LNI 192.168.203.107@tcp [ 7895.261068] Key type .llcrypt unregistered [ 7895.265378] Key type ._llcrypt unregistered [ 7912.548659] Key type ._llcrypt registered [ 7912.550647] Key type .llcrypt registered [ 7912.655791] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 7913.741627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7913.775602] alg: No test for adler32 (adler32-zlib) [ 7914.782394] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7915.060884] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 7916.695171] Key type lgssc registered [ 7917.748593] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7925.822455] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7925.847550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7927.410413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7932.631949] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7938.017207] LustreError: 267471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7938.042115] LustreError: 267471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7940.433178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7940.816274] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7945.258706] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7952.816589] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7958.146275] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7964.790828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7965.231688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7970.945354] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 7972.416250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7977.326705] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7991.776919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7991.784461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7991.795537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7992.800483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7992.802318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7992.804130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7994.737910] Lustre: server umount lustre-OST0000 complete [ 8002.531131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8002.540522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8002.557341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8002.567168] Lustre: Skipped 1 previous similar message [ 8004.571498] Lustre: server umount lustre-MDT0000 complete [ 8008.145874] LustreError: 267452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161016 with bad export cookie 15380120498269852655 [ 8008.149856] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8008.154060] LustreError: 267452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8008.495449] Lustre: server umount lustre-MDT0001 complete [ 8019.694729] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 8022.336696] Key type lgssc unregistered [ 8022.628380] LNet: 271148:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8022.644267] LNetError: 271148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8022.663990] LNet: Removed LNI 192.168.203.107@tcp [ 8023.476657] Key type .llcrypt unregistered [ 8023.480186] Key type ._llcrypt unregistered [ 8045.140606] Key type ._llcrypt registered [ 8045.143267] Key type .llcrypt registered [ 8045.275277] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 8046.447318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8046.489893] alg: No test for adler32 (adler32-zlib) [ 8047.514488] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8047.710071] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 8049.383143] Key type lgssc registered [ 8050.447144] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8057.206278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8057.223607] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8058.652681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8062.872930] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8069.090725] LustreError: 272564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8069.105269] LustreError: 272564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8070.906427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8071.310507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8075.582190] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8083.573606] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8089.483706] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8096.230578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8096.588849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8102.929132] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8103.731916] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8112.277645] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8139.747239] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8139.757591] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.772137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8141.805095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8141.812360] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8141.818808] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8141.863478] Lustre: Skipped 1 previous similar message [ 8145.468845] Lustre: server umount lustre-OST0000 complete [ 8152.032745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8152.039250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8152.052727] Lustre: Skipped 1 previous similar message [ 8152.075909] Lustre: Skipped 1 previous similar message [ 8155.270332] Lustre: server umount lustre-MDT0000 complete [ 8157.154807] LustreError: 272565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8157.161531] LustreError: 272565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8158.972420] LustreError: 275834:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161167 with bad export cookie 9922008072560475457 [ 8158.985757] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8159.292142] Lustre: server umount lustre-MDT0001 complete [ 8172.087123] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 8174.951505] Key type lgssc unregistered [ 8175.272239] LNet: 276875:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8175.278631] LNetError: 276875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8175.297097] LNet: Removed LNI 192.168.203.107@tcp [ 8176.231328] Key type .llcrypt unregistered [ 8176.236331] Key type ._llcrypt unregistered [ 8193.971962] Key type ._llcrypt registered [ 8193.974174] Key type .llcrypt registered [ 8194.100523] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 8194.782884] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8194.794288] alg: No test for adler32 (adler32-zlib) [ 8195.974765] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8196.277397] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 8197.935171] Key type lgssc registered [ 8199.432611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8207.098876] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8207.112862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8208.733825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8213.434513] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8219.109407] LustreError: 278111:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8219.129466] LustreError: 278111:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8221.455605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.888776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8225.881655] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8232.645398] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8238.676885] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8247.312317] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8247.757642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8253.962652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8256.864875] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8266.223479] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8279.521569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8279.540182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8279.560530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8280.059719] Lustre: server umount lustre-OST0000 complete [ 8287.715178] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8287.723451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8287.725978] Lustre: Skipped 1 previous similar message [ 8287.739444] Lustre: Skipped 1 previous similar message [ 8290.228578] Lustre: server umount lustre-MDT0000 complete [ 8292.835057] LustreError: 278114:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8292.869149] LustreError: 278114:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8294.011279] LustreError: 278077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161302 with bad export cookie 6744501676911017633 [ 8294.016995] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8294.038862] LustreError: 278077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8303.754083] Lustre: server umount lustre-MDT0001 complete [ 8316.100661] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 8318.647198] Key type lgssc unregistered [ 8318.909695] LNet: 281813:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8318.925069] LNetError: 281813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8318.946765] LNet: Removed LNI 192.168.203.107@tcp [ 8319.777955] Key type .llcrypt unregistered [ 8319.781290] Key type ._llcrypt unregistered [ 8330.582311] Key type ._llcrypt registered [ 8330.584725] Key type .llcrypt registered [ 8330.754285] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8348.435428] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 8364.816315] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 8365.595622] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8365.745619] alg: No test for adler32 (adler32-zlib) [ 8367.001948] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8367.350991] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 8369.135277] Key type lgssc registered [ 8370.885631] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8410.736701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8410.792622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8412.077845] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8412.125411] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8412.258852] Lustre: lustre-MDT0000: new disk, initializing [ 8412.400273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.418712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8417.508566] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8429.033418] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8429.220570] Lustre: 285368:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8429.283708] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8429.290417] Lustre: Skipped 1 previous similar message [ 8429.396542] Lustre: lustre-MDT0001: new disk, initializing [ 8429.492220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8429.545749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8429.560596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8434.464694] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8443.731316] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8449.282456] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8455.537775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8455.886595] Lustre: lustre-OST0000: new disk, initializing [ 8455.892683] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8455.902101] Lustre: 287341:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8455.992869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8462.265331] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8463.441504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8463.450599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8463.611182] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8472.389736] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8477.754114] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8478.008382] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8483.104150] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8483.402907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8489.441585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8489.452531] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8489.460294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8489.464227] Lustre: Skipped 1 previous similar message [ 8489.478475] Lustre: Skipped 1 previous similar message [ 8494.561168] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8494.583603] Lustre: Skipped 1 previous similar message [ 8499.695611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8499.705579] Lustre: Skipped 1 previous similar message [ 8500.191152] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8500.397694] Lustre: server umount lustre-OST0000 complete [ 8504.804392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.809955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8504.820163] Lustre: Skipped 1 previous similar message [ 8504.845562] Lustre: Skipped 1 previous similar message [ 8509.922079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8509.937032] Lustre: Skipped 1 previous similar message [ 8519.135602] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8519.320941] Lustre: server umount lustre-MDT0000 complete [ 8523.362040] LustreError: 286167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161532 with bad export cookie 6797881446340510344 [ 8523.365498] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8523.374967] LustreError: 286167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8523.880153] Lustre: server umount lustre-MDT0001 complete [ 8533.108371] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:32:20 (1788161540) [ 8540.443789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8541.005147] LustreError: 289754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8541.022529] LustreError: 289754:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 8541.088901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8546.281515] LustreError: 289755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8546.303660] LustreError: 289755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8546.473931] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8550.386127] LustreError: 289754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8555.289837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8555.499053] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 8555.503597] Lustre: Skipped 2 previous similar messages [ 8555.672307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8560.637628] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8568.814851] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8574.836351] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8582.061141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8582.358739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8588.964723] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8597.094507] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8615.308597] Lustre: server umount lustre-OST0000 complete [ 8622.048387] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8622.069159] LustreError: Skipped 1 previous similar message [ 8622.075492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8622.085447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8629.838833] Lustre: server umount lustre-MDT0000 complete [ 8632.824430] LustreError: 289755:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8632.852114] LustreError: 289755:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8633.887191] Lustre: 283594:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788161626/real 1788161626] req@000000002c2e2198 x1875023103076224/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788161642 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8633.920746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8633.933174] Lustre: Skipped 1 previous similar message [ 8634.302469] LustreError: 289736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161643 with bad export cookie 6797881446340511226 [ 8634.308560] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8634.318651] LustreError: 289736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8634.820444] Lustre: server umount lustre-MDT0001 complete [ 8646.597817] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 8649.476222] Key type lgssc unregistered [ 8649.864137] LNet: 293310:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8649.871804] LNetError: 293310:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8649.907161] LNet: Removed LNI 192.168.203.107@tcp [ 8650.816925] Key type .llcrypt unregistered [ 8650.818558] Key type ._llcrypt unregistered [ 8678.328258] Key type ._llcrypt registered [ 8678.331414] Key type .llcrypt registered [ 8678.440302] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 8692.880584] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 8694.129820] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8694.166474] alg: No test for adler32 (adler32-zlib) [ 8695.310961] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8695.678049] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 8697.327177] Key type lgssc registered [ 8698.379220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8742.101920] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8742.146445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8743.305099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8743.331963] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8743.394902] Lustre: lustre-MDT0000: new disk, initializing [ 8743.470814] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8743.485699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8749.070658] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8759.038408] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8759.139698] Lustre: 297137:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8759.195256] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8759.204960] Lustre: Skipped 1 previous similar message [ 8759.310513] Lustre: lustre-MDT0001: new disk, initializing [ 8759.388434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8759.426055] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8759.433936] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8763.445239] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8771.913445] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8777.687208] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8784.295955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8784.562182] Lustre: lustre-OST2710: new disk, initializing [ 8784.565690] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8784.572311] Lustre: 299106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8784.674807] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8790.460597] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8794.659858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8794.679067] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8794.854387] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8799.884084] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8807.859412] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8808.036958] Lustre: lustre-OST03e8: new disk, initializing [ 8808.040874] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8808.047449] Lustre: 300170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8808.117766] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8813.669985] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8813.685975] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8813.820777] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8814.385366] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8821.534235] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8828.726895] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8828.892480] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8833.972821] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8834.200799] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8838.659163] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8838.856988] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8842.872671] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8843.056430] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8849.538878] Lustre: server umount lustre-MDT0000 complete [ 8849.899296] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8849.920947] Lustre: Skipped 1 previous similar message [ 8852.974485] LustreError: 297130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161861 with bad export cookie 6449706861243674011 [ 8852.977262] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8852.989937] LustreError: 297130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8853.439925] Lustre: server umount lustre-MDT0001 complete [ 8867.409948] Lustre: server umount lustre-OST2710 complete [ 8870.310338] Lustre: 294937:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788161863/real 1788161863] req@00000000ab50df02 x1875023446999040/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788161879 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8870.328408] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8871.210552] Lustre: server umount lustre-OST03e8 complete [ 8889.332399] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [ 8898.649641] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 8936.569050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8936.982955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8937.030622] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8937.178687] Lustre: lustre-MDT0000: new disk, initializing [ 8937.323200] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8937.355258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8942.007037] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8953.734027] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8953.890033] Lustre: 305414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8953.980566] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8953.985847] Lustre: Skipped 1 previous similar message [ 8954.086779] Lustre: lustre-MDT0001: new disk, initializing [ 8954.233625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8954.282919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8954.302268] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8959.293960] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8970.007354] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8976.763370] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8985.049314] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8985.529301] Lustre: lustre-OST0000: new disk, initializing [ 8985.532715] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8985.537559] Lustre: 307384:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8985.605113] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8991.066983] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 8995.404991] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8995.423032] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8995.535380] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9001.256657] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9005.835210] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9006.010486] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9010.948808] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9011.216553] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9016.306132] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9016.318750] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9016.329996] Lustre: Skipped 1 previous similar message [ 9016.330706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9016.349162] Lustre: Skipped 1 previous similar message [ 9019.583864] Lustre: server umount lustre-OST0000 complete [ 9026.018256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9026.026063] LustreError: Skipped 1 previous similar message [ 9026.035668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9026.058028] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9030.009417] Lustre: server umount lustre-MDT0000 complete [ 9031.651411] LustreError: 305428:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9031.666031] LustreError: 305428:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 9034.013094] LustreError: 305407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162042 with bad export cookie 6449706861243675691 [ 9034.030375] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9034.032493] LustreError: 305407:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9034.470103] Lustre: server umount lustre-MDT0001 complete [ 9042.923589] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:40:50 (1788162050) [ 9044.295679] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9046.049457] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:40:53 (1788162053) [ 9049.225750] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 9079.066537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9079.525695] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9079.552473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9079.752273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9084.475321] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9095.503890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9095.677763] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9095.704577] Lustre: 311243:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9101.197936] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9123.316595] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9123.510990] Lustre: Denying initial registration attempt from nid 192.168.203.107@tcp, specified as failover [ 9123.522489] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9123.534298] LustreError: 312700:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -99 [ 9123.545313] LustreError: 312700:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 9123.560703] LustreError: 312700:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9123.632688] Lustre: server umount lustre-OST0000 complete [ 9123.635046] LustreError: 312700:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9132.000161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9132.007038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9132.034069] Lustre: Skipped 1 previous similar message [ 9132.047899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9132.054814] Lustre: Skipped 1 previous similar message [ 9135.586885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9135.597692] Lustre: Skipped 1 previous similar message [ 9137.754448] Lustre: server umount lustre-MDT0000 complete [ 9140.708472] LustreError: 311252:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9140.748833] LustreError: 311252:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9141.516129] LustreError: 311236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162150 with bad export cookie 6449706861243676566 [ 9141.535677] LustreError: 311236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9141.542688] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9141.906821] Lustre: server umount lustre-MDT0001 complete [ 9153.181840] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 9156.445477] Key type lgssc unregistered [ 9156.704316] LNet: 313819:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9156.715358] LNetError: 313819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9156.734550] LNet: Removed LNI 192.168.203.107@tcp [ 9157.856849] Key type .llcrypt unregistered [ 9157.863778] Key type ._llcrypt unregistered [ 9173.699753] Key type ._llcrypt registered [ 9173.707728] Key type .llcrypt registered [ 9173.842603] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 9174.844570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9174.877157] alg: No test for adler32 (adler32-zlib) [ 9175.912182] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9176.099019] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 9177.927254] Key type lgssc registered [ 9179.124793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9225.564760] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 9232.903166] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9232.925508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9234.076244] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9234.088972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9234.185590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9238.410122] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9245.775431] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.850934] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9245.887109] Lustre: 316586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9246.140568] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9250.350611] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9257.461326] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9262.821445] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9282.918350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9283.193206] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9283.462746] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9289.485362] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9296.604503] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9303.016335] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9303.030144] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9303.055406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9303.524483] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9303.534289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9304.861317] Lustre: server umount lustre-OST0000 complete [ 9308.642113] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9308.652697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9308.654502] Lustre: Skipped 1 previous similar message [ 9308.663891] Lustre: Skipped 2 previous similar messages [ 9313.761663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9313.771346] Lustre: Skipped 1 previous similar message [ 9314.539668] Lustre: server umount lustre-MDT0000 complete [ 9318.311603] LustreError: 316576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162327 with bad export cookie 4530260763006982432 [ 9318.322763] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9318.329212] LustreError: 316576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9318.674389] Lustre: server umount lustre-MDT0001 complete [ 9328.781328] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 9331.179432] Key type lgssc unregistered [ 9331.428047] LNet: 320156:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9331.436405] LNetError: 320156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9331.469248] LNet: Removed LNI 192.168.203.107@tcp [ 9332.290992] Key type .llcrypt unregistered [ 9332.294479] Key type ._llcrypt unregistered [ 9350.893807] Key type ._llcrypt registered [ 9350.896786] Key type .llcrypt registered [ 9350.998970] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 9351.304515] Key type .llcrypt unregistered [ 9351.309689] Key type ._llcrypt unregistered [ 9365.692107] Key type ._llcrypt registered [ 9365.694691] Key type .llcrypt registered [ 9365.832413] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 9367.044754] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9367.063205] alg: No test for adler32 (adler32-zlib) [ 9368.083724] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9368.293030] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 9369.994685] Key type lgssc registered [ 9371.180901] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9403.223889] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [ 9404.403860] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [ 9417.580089] loop0: detected capacity change from 0 to 204800000 [ 9417.724935] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9417.751341] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9417.762771] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9434.411501] loop0: detected capacity change from 0 to 204800000 [ 9434.624947] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9434.643965] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9434.654297] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9452.735953] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9452.744664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9453.934337] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9453.947277] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9454.058481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9458.491526] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9467.707730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9467.814856] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9467.849481] Lustre: 324062:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9468.111741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9473.199287] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9480.924178] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9487.381277] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9492.891397] loop0: detected capacity change from 0 to 204800000 [ 9511.278986] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9511.659992] Lustre: sv27749-OST0000: new disk, initializing [ 9511.668497] Lustre: srv-sv27749-OST0000: No data found on store. Initialize space. [ 9511.684152] Lustre: 326143:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9511.778592] Lustre: sv27749-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9519.210161] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9529.473881] loop1: detected capacity change from 0 to 204800000 [ 9547.814307] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9547.959630] Lustre: sv27749-OST0001: new disk, initializing [ 9547.962236] Lustre: srv-sv27749-OST0001: No data found on store. Initialize space. [ 9547.968402] Lustre: 327261:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9548.026294] Lustre: sv27749-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9554.265689] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9567.179605] Lustre: server umount sv27749-OST0001 complete [ 9578.148971] Lustre: server umount sv27749-OST0000 complete [ 9600.492271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9600.498381] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9600.507949] Lustre: Skipped 1 previous similar message [ 9603.865525] Lustre: server umount lustre-MDT0000 complete [ 9607.371107] LustreError: 324057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162616 with bad export cookie 17972142218905761278 [ 9607.374408] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9607.387300] LustreError: 324057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9607.675816] Lustre: server umount lustre-MDT0001 complete [ 9619.097598] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 9622.087226] Key type lgssc unregistered [ 9622.466744] LNet: 330014:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9622.474540] LNetError: 330014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9623.519335] LNet: Removed LNI 192.168.203.107@tcp [ 9623.522509] LNetError: 330014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9623.536954] LNet: Removed LNI 192.168.203.107@tcp1 [ 9624.722607] Key type .llcrypt unregistered [ 9624.724946] Key type ._llcrypt unregistered [ 9646.230342] Key type ._llcrypt registered [ 9646.232329] Key type .llcrypt registered [ 9646.325153] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 9646.571668] Key type .llcrypt unregistered [ 9646.573719] Key type ._llcrypt unregistered [ 9664.276629] Key type ._llcrypt registered [ 9664.280687] Key type .llcrypt registered [ 9664.454625] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 9665.290322] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9665.316395] alg: No test for adler32 (adler32-zlib) [ 9666.617504] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9666.811793] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 9668.495165] Key type lgssc registered [ 9669.673269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9689.066855] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [ 9690.274276] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [ 9699.179840] loop0: detected capacity change from 0 to 204800000 [ 9699.325762] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9699.338402] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9699.363633] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9715.838467] loop0: detected capacity change from 0 to 204800000 [ 9715.981375] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9716.006548] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9716.022460] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9733.249136] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9733.269169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9734.506905] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9734.529603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9734.705849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9738.929253] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9747.135916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.252136] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9747.285894] Lustre: 333837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9747.569892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9752.392039] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9759.239449] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9764.154529] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9768.885427] loop0: detected capacity change from 0 to 204800000 [ 9786.916987] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9787.138517] Lustre: sv12896-OST0000: new disk, initializing [ 9787.148027] Lustre: srv-sv12896-OST0000: No data found on store. Initialize space. [ 9787.160700] Lustre: 335913:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9787.233202] Lustre: sv12896-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.042461] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9802.369356] loop1: detected capacity change from 0 to 204800000 [ 9821.997690] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9822.175812] Lustre: sv12896-OST0001: new disk, initializing [ 9822.180802] Lustre: srv-sv12896-OST0001: No data found on store. Initialize space. [ 9822.192787] Lustre: 337031:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9822.265277] Lustre: sv12896-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9830.882908] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [ 9845.908344] Lustre: server umount sv12896-OST0001 complete [ 9858.499569] Lustre: server umount sv12896-OST0000 complete [ 9875.430392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9875.441534] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9875.467728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.009549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9879.015637] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9879.059913] Lustre: Skipped 1 previous similar message [ 9881.353891] Lustre: server umount lustre-MDT0000 complete [ 9885.030177] LustreError: 337030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162893 with bad export cookie 17244120507835297457 [ 9885.047620] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9885.051363] LustreError: 337030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9885.711892] Lustre: server umount lustre-MDT0001 complete [ 9900.236784] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 9903.662095] Key type lgssc unregistered [ 9903.945580] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.7@tcp [ 9903.997860] LNet: 339789:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9904.012647] LNetError: 339789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9905.056222] LNet: Removed LNI 192.168.203.107@tcp [ 9905.060949] LNetError: 339789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9906.086579] LNet: Removed LNI 192.168.203.107@tcp1 [ 9906.873299] Key type .llcrypt unregistered [ 9906.877859] Key type ._llcrypt unregistered [ 9925.406260] Key type ._llcrypt registered [ 9925.408573] Key type .llcrypt registered [ 9925.509536] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [ 9925.919743] Key type .llcrypt unregistered [ 9925.925155] Key type ._llcrypt unregistered [ 9942.204191] Key type ._llcrypt registered [ 9942.208760] Key type .llcrypt registered [ 9942.361208] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [ 9943.672737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9943.754035] alg: No test for adler32 (adler32-zlib) [ 9944.868426] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9945.115588] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [ 9946.815651] Key type lgssc registered [ 9948.083845] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9971.960537] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [ 9973.333175] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [ 9988.101936] loop0: detected capacity change from 0 to 204800000 [ 9988.399869] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9988.421966] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 9988.435887] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10006.195634] loop0: detected capacity change from 0 to 204800000 [10006.401713] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10006.409497] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10006.416855] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10024.523845] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10024.549550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.841835] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10025.870601] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10026.126437] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10030.659697] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10040.086063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10040.271372] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10040.292031] Lustre: 343611:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10040.564332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10045.428966] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10054.045491] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10059.350539] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10064.709398] loop0: detected capacity change from 0 to 204800000 [10082.524673] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10082.957335] Lustre: sv30290-OST0000: new disk, initializing [10082.967756] Lustre: srv-sv30290-OST0000: No data found on store. Initialize space. [10082.984710] Lustre: 345692:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10083.085208] Lustre: sv30290-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10090.174702] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10110.366315] Lustre: server umount sv30290-OST0000 complete [10127.842532] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10127.846867] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10127.869447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10127.894100] Lustre: Skipped 1 previous similar message [10129.633190] Lustre: server umount lustre-MDT0000 complete [10133.332757] LustreError: 343606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163142 with bad export cookie 13201213165439354726 [10133.334881] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10133.344382] LustreError: 343606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10133.641856] Lustre: server umount lustre-MDT0001 complete [10145.674289] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [10148.841690] Key type lgssc unregistered [10149.110276] LNet: 348144:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10149.120552] LNetError: 348144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10149.132061] LNet: Removed LNI 192.168.203.107@tcp [10150.182192] LNet: Removed LNI 192.168.203.107@tcp1 [10151.249940] Key type .llcrypt unregistered [10151.256193] Key type ._llcrypt unregistered [10171.587710] Key type ._llcrypt registered [10171.592204] Key type .llcrypt registered [10171.809397] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [10172.590434] Key type .llcrypt unregistered [10172.596157] Key type ._llcrypt unregistered [10192.115079] Key type ._llcrypt registered [10192.121889] Key type .llcrypt registered [10192.229758] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [10193.526080] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10193.569062] alg: No test for adler32 (adler32-zlib) [10194.564470] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10194.843076] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [10196.567147] Key type lgssc registered [10197.874349] Lustre: Echo OBD driver; http://www.lustre.org/ [10221.394955] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [10223.251716] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [10235.758782] loop0: detected capacity change from 0 to 204800000 [10235.932808] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10235.943041] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10235.950227] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10254.506386] loop0: detected capacity change from 0 to 204800000 [10254.809455] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10254.825499] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10254.835144] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10273.521620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10273.551729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10274.905288] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10274.929355] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10275.125105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10280.244275] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10290.141200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10290.353613] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10290.403741] Lustre: 352027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10290.897676] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10296.588392] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10305.495651] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10312.706538] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10319.067232] loop0: detected capacity change from 0 to 204800000 [10339.061428] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10339.479637] Lustre: sv3576-OST0000: new disk, initializing [10339.484813] Lustre: srv-sv3576-OST0000: No data found on store. Initialize space. [10339.494706] Lustre: 354106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10339.576447] Lustre: sv3576-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10347.377661] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10364.637278] Lustre: server umount sv3576-OST0000 complete [10377.696841] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10377.706278] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10377.723182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10379.745767] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10379.747536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10383.719478] Lustre: server umount lustre-MDT0000 complete [10387.673730] LustreError: 352020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163396 with bad export cookie 1091929662259797263 [10387.677203] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10387.687019] LustreError: 352020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10388.087524] Lustre: server umount lustre-MDT0001 complete [10400.689559] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [10403.674198] Key type lgssc unregistered [10404.023633] LNet: 356558:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10404.035708] LNetError: 356558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10404.052557] LNet: Removed LNI 192.168.203.107@tcp [10405.095915] LNet: Removed LNI 192.168.203.107@tcp1 [10406.274328] Key type .llcrypt unregistered [10406.277279] Key type ._llcrypt unregistered [10425.023391] Key type ._llcrypt registered [10425.025644] Key type .llcrypt registered [10425.142564] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [10425.382453] Key type .llcrypt unregistered [10425.388743] Key type ._llcrypt unregistered [10440.889896] Key type ._llcrypt registered [10440.892761] Key type .llcrypt registered [10441.002613] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [10442.064449] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10442.197934] alg: No test for adler32 (adler32-zlib) [10443.334901] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10443.643694] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [10445.375171] Key type lgssc registered [10446.762851] Lustre: Echo OBD driver; http://www.lustre.org/ [10468.336790] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [10469.852260] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [10480.853739] loop0: detected capacity change from 0 to 204800000 [10481.117467] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10481.132504] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10481.148743] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10498.707667] loop0: detected capacity change from 0 to 204800000 [10498.898426] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10498.909439] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10498.922704] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10517.191206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10517.222030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10518.541459] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10518.575115] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10518.706238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10522.786242] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10531.090376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10531.222917] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10531.289403] Lustre: 360380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10531.557266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10535.950920] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10543.171698] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10549.430764] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10557.073691] loop0: detected capacity change from 0 to 204800000 [10578.050741] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10578.375551] Lustre: sv27517-OST0000: new disk, initializing [10578.378508] Lustre: srv-sv27517-OST0000: No data found on store. Initialize space. [10578.385629] Lustre: 362459:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10578.450693] Lustre: sv27517-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10586.081466] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10600.625222] loop1: detected capacity change from 0 to 204800000 [10620.076619] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10620.239277] Lustre: sv27517-OST0001: new disk, initializing [10620.242640] Lustre: srv-sv27517-OST0001: No data found on store. Initialize space. [10620.248788] Lustre: 363629:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10620.310787] Lustre: sv27517-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10626.541734] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10638.826289] Lustre: server umount sv27517-OST0001 complete [10648.011555] Lustre: server umount sv27517-OST0000 complete [10659.808120] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10659.821245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10659.847748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10661.857374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10661.858819] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10664.950215] Lustre: server umount lustre-MDT0000 complete [10668.190147] LustreError: 360375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163676 with bad export cookie 4474582734794259436 [10668.209112] LustreError: 360375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10668.209545] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10668.494432] Lustre: server umount lustre-MDT0001 complete [10679.728398] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [10682.657209] Key type lgssc unregistered [10682.927743] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.7@tcp [10682.954676] LNet: 366382:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10682.964817] LNetError: 366382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10683.999194] LNet: Removed LNI 192.168.203.107@tcp [10684.001202] LNetError: 366382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10684.054285] LNet: Removed LNI 192.168.203.107@tcp1 [10684.970477] Key type .llcrypt unregistered [10684.972394] Key type ._llcrypt unregistered [10703.176542] Key type ._llcrypt registered [10703.181761] Key type .llcrypt registered [10703.356699] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [10703.638894] Key type .llcrypt unregistered [10703.641929] Key type ._llcrypt unregistered [10722.097230] Key type ._llcrypt registered [10722.101281] Key type .llcrypt registered [10722.183552] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [10723.119996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10723.134619] alg: No test for adler32 (adler32-zlib) [10724.176258] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10724.417995] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [10726.090575] Key type lgssc registered [10727.073295] Lustre: Echo OBD driver; http://www.lustre.org/ [10747.101640] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [10748.246771] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [10776.832155] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [10779.358717] Key type lgssc unregistered [10779.587701] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.7@tcp [10779.612613] LNet: 370345:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10779.618075] LNetError: 370345:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10779.625984] LNet: Removed LNI 192.168.203.107@tcp [10779.643960] LNet: Removed LNI 192.168.203.107@tcp1 [10780.328083] Key type .llcrypt unregistered [10780.330431] Key type ._llcrypt unregistered [10797.593834] Key type ._llcrypt registered [10797.596570] Key type .llcrypt registered [10797.690455] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [10798.008630] Key type .llcrypt unregistered [10798.012293] Key type ._llcrypt unregistered [10814.951099] Key type ._llcrypt registered [10814.954798] Key type .llcrypt registered [10815.047063] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [10816.051044] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10816.071458] alg: No test for adler32 (adler32-zlib) [10817.117401] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10817.319372] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [10818.943742] Key type lgssc registered [10820.182217] Lustre: Echo OBD driver; http://www.lustre.org/ [10840.246728] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [10841.253109] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [10843.127739] LNet: Added LNI 192.168.203.107@tcp2 [8/256/0/180] [10852.671534] loop0: detected capacity change from 0 to 204800000 [10852.799851] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10852.810752] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10852.821638] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10869.733440] loop0: detected capacity change from 0 to 204800000 [10869.921536] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10869.930867] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10869.944996] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10887.769939] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10887.793535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10889.078677] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10889.119501] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10889.256446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10889.259682] LNetError: 371672:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.203.107@tcp NID 192.168.203.107@tcp is already locked with peer 192.168.203.107@tcp [10893.927677] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10902.871161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10903.006900] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10903.099821] Lustre: 374257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10903.316546] LNetError: 371672:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.203.107@tcp NID 192.168.203.107@tcp is already locked with peer 192.168.203.107@tcp [10903.337780] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10903.464642] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10908.033443] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10915.893506] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10922.128024] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10927.457461] loop0: detected capacity change from 0 to 204800000 [10946.354134] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10946.677899] Lustre: sv3554-OST0000: new disk, initializing [10946.686133] Lustre: srv-sv3554-OST0000: No data found on store. Initialize space. [10946.697346] Lustre: 376337:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10946.757355] Lustre: sv3554-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10953.025498] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [10963.892723] loop1: detected capacity change from 0 to 204800000 [10982.694191] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10982.843378] Lustre: sv3554-OST0001: new disk, initializing [10982.847589] Lustre: srv-sv3554-OST0001: No data found on store. Initialize space. [10982.855604] Lustre: 377505:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10982.938468] Lustre: sv3554-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10990.359777] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11005.232845] Lustre: server umount sv3554-OST0001 complete [11016.945492] Lustre: server umount sv3554-OST0000 complete [11036.644176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11036.650607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11036.651833] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11036.704528] Lustre: Skipped 1 previous similar message [11038.978296] Lustre: server umount lustre-MDT0000 complete [11042.680759] LustreError: 374251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164051 with bad export cookie 17407420913674323098 [11042.681490] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11042.692990] LustreError: 374251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11043.192858] Lustre: server umount lustre-MDT0001 complete [11056.256249] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11059.059802] Key type lgssc unregistered [11059.354919] LNet: 380260:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11059.361845] LNetError: 380260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11059.370154] LNet: Removed LNI 192.168.203.107@tcp [11060.383249] LNet: Removed LNI 192.168.203.107@tcp1 [11060.385270] LNetError: 380260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11060.400407] LNetError: 380260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11060.417494] LNet: Removed LNI 192.168.203.107@tcp2 [11061.290264] Key type .llcrypt unregistered [11061.295393] Key type ._llcrypt unregistered [11080.382819] Key type ._llcrypt registered [11080.385123] Key type .llcrypt registered [11080.462263] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11080.702362] Key type .llcrypt unregistered [11080.709351] Key type ._llcrypt unregistered [11096.284934] Key type ._llcrypt registered [11096.287585] Key type .llcrypt registered [11096.372698] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [11097.656912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11097.675222] alg: No test for adler32 (adler32-zlib) [11098.742926] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11099.043498] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [11100.703179] Key type lgssc registered [11101.650821] Lustre: Echo OBD driver; http://www.lustre.org/ [11120.581602] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [11121.881767] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [11131.971150] loop0: detected capacity change from 0 to 204800000 [11132.253759] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11132.274927] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11132.288605] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11149.541502] loop0: detected capacity change from 0 to 204800000 [11149.734438] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11149.760714] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11149.777904] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11169.239915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11169.295674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11170.544039] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11170.556313] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11170.650469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11175.027671] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11183.821821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11183.884155] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11183.905953] Lustre: 384133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11184.166459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11188.852251] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11196.498625] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11202.018408] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11206.612962] loop0: detected capacity change from 0 to 204800000 [11225.059861] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11225.372707] Lustre: sv22326-OST0000: new disk, initializing [11225.386206] Lustre: srv-sv22326-OST0000: No data found on store. Initialize space. [11225.395485] Lustre: 386208:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11225.457432] Lustre: sv22326-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11231.510094] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11240.637230] loop1: detected capacity change from 0 to 204800000 [11259.371610] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11259.543366] Lustre: sv22326-OST0001: new disk, initializing [11259.548165] Lustre: srv-sv22326-OST0001: No data found on store. Initialize space. [11259.554702] Lustre: 387327:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11259.606677] Lustre: sv22326-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11265.521950] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11279.000075] Lustre: server umount sv22326-OST0001 complete [11286.675392] Lustre: server umount sv22326-OST0000 complete [11301.866707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11301.868627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11301.877179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11301.888842] Lustre: Skipped 1 previous similar message [11304.317594] Lustre: server umount lustre-MDT0000 complete [11307.621899] LustreError: 384123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164316 with bad export cookie 14634200185314525709 [11307.626744] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11307.631063] LustreError: 384123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11307.894341] Lustre: server umount lustre-MDT0001 complete [11320.464457] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11323.005600] Key type lgssc unregistered [11323.316736] LNet: 1 peer NIs in recovery (showing 1): 192.168.203.7@tcp [11323.357126] LNet: 390081:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11323.361774] LNetError: 390081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11323.370259] LNet: Removed LNI 192.168.203.107@tcp [11324.389751] LNet: Removed LNI 192.168.203.107@tcp1 [11325.207324] Key type .llcrypt unregistered [11325.213309] Key type ._llcrypt unregistered [11342.705644] Key type ._llcrypt registered [11342.708717] Key type .llcrypt registered [11342.819861] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11343.076306] Key type .llcrypt unregistered [11343.080700] Key type ._llcrypt unregistered [11348.942804] Key type ._llcrypt registered [11348.945978] Key type .llcrypt registered [11349.057955] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11349.348515] Key type .llcrypt unregistered [11349.354277] Key type ._llcrypt unregistered [11363.330972] Key type ._llcrypt registered [11363.333022] Key type .llcrypt registered [11363.419911] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11363.706380] Key type .llcrypt unregistered [11363.710293] Key type ._llcrypt unregistered [11383.932676] Key type ._llcrypt registered [11383.935597] Key type .llcrypt registered [11384.068403] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11384.401411] Key type .llcrypt unregistered [11384.407059] Key type ._llcrypt unregistered [11403.551305] Key type ._llcrypt registered [11403.559168] Key type .llcrypt registered [11403.711838] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [11404.590571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11404.809463] alg: No test for adler32 (adler32-zlib) [11405.739864] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11405.927187] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [11407.535182] Key type lgssc registered [11408.646521] Lustre: Echo OBD driver; http://www.lustre.org/ [11427.649757] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [11428.847270] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [11440.548668] loop0: detected capacity change from 0 to 204800000 [11440.786533] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11440.807915] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11440.823387] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11458.067653] loop0: detected capacity change from 0 to 204800000 [11458.338885] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11458.348270] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11458.377312] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11475.323616] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11475.337188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11476.581465] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11476.608507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11476.785654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11480.350623] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11487.795592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11487.878879] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11487.911514] Lustre: 395571:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11488.116580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11492.000650] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11499.208775] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11504.282318] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11509.056382] loop0: detected capacity change from 0 to 204800000 [11526.959466] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11527.299324] Lustre: sv27149-OST0000: new disk, initializing [11527.304689] Lustre: srv-sv27149-OST0000: No data found on store. Initialize space. [11527.309361] Lustre: 397648:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11527.406459] Lustre: sv27149-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11534.910931] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11546.405885] loop1: detected capacity change from 0 to 204800000 [11566.889672] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11567.085676] Lustre: sv27149-OST0001: new disk, initializing [11567.094972] Lustre: srv-sv27149-OST0001: No data found on store. Initialize space. [11567.107086] Lustre: 398766:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11567.182049] Lustre: sv27149-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11573.291480] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11589.687592] Lustre: 399756:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [11589.695789] Lustre: 399756:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [11589.703349] Lustre: 399756:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [11589.709798] Lustre: 399756:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [11589.717107] Lustre: 399756:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [11589.728891] Lustre: 399756:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [11592.257969] Lustre: server umount sv27149-OST0001 complete [11602.570867] Lustre: server umount sv27149-OST0000 complete [11616.229481] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11616.234817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11616.251042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11617.760093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11617.760278] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11622.330752] Lustre: server umount lustre-MDT0000 complete [11627.577114] LustreError: 395565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164636 with bad export cookie 7163798650590166882 [11627.583158] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11627.592026] LustreError: 395565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11627.865664] Lustre: server umount lustre-MDT0001 complete [11642.807933] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11645.959114] Key type lgssc unregistered [11646.422727] LNet: 401761:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11646.429887] LNetError: 401761:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11646.439990] LNet: Removed LNI 192.168.203.107@tcp [11647.465078] LNet: Removed LNI 192.168.203.107@tcp1 [11649.063715] Key type .llcrypt unregistered [11649.066177] Key type ._llcrypt unregistered [11668.787299] Key type ._llcrypt registered [11668.789627] Key type .llcrypt registered [11668.929474] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11669.373684] Key type .llcrypt unregistered [11669.381807] Key type ._llcrypt unregistered [11683.665923] Key type ._llcrypt registered [11683.669279] Key type .llcrypt registered [11683.807725] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [11684.882889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11684.906710] alg: No test for adler32 (adler32-zlib) [11686.013218] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11686.252427] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [11687.935994] Key type lgssc registered [11688.924716] Lustre: Echo OBD driver; http://www.lustre.org/ [11708.853887] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [11709.931050] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [11720.819480] loop0: detected capacity change from 0 to 204800000 [11721.030790] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11721.048156] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11721.057321] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11739.002607] loop0: detected capacity change from 0 to 204800000 [11739.176732] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11739.193667] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11739.206955] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11758.482941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11758.500953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11759.719488] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11759.747264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11759.951829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11764.551110] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11773.245672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11773.359819] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11773.404590] Lustre: 405566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11773.670995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11777.925509] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11785.202349] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11790.720932] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11795.435032] loop0: detected capacity change from 0 to 204800000 [11814.060805] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11814.335214] Lustre: sv3098-OST0000: new disk, initializing [11814.344629] Lustre: srv-sv3098-OST0000: No data found on store. Initialize space. [11814.357857] Lustre: 407643:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11814.415289] Lustre: sv3098-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11820.480793] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11830.542897] loop1: detected capacity change from 0 to 204800000 [11848.748433] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11848.925994] Lustre: sv3098-OST0001: new disk, initializing [11848.931600] Lustre: srv-sv3098-OST0001: No data found on store. Initialize space. [11848.938955] Lustre: 408761:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11848.989107] Lustre: sv3098-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11854.982622] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [11870.642377] Lustre: 409754:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [11870.648214] Lustre: 409754:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [11870.662029] Lustre: 409754:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [11870.676287] Lustre: 409754:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [11870.692185] Lustre: 409754:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [11870.698212] Lustre: 409754:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [11872.907466] Lustre: server umount sv3098-OST0001 complete [11883.138614] Lustre: server umount sv3098-OST0000 complete [11898.338969] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11898.353638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11898.365215] Lustre: Skipped 1 previous similar message [11898.385237] Lustre: Skipped 1 previous similar message [11903.457016] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11903.473636] Lustre: Skipped 1 previous similar message [11904.613579] Lustre: server umount lustre-MDT0000 complete [11909.131319] LustreError: 405556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164917 with bad export cookie 16130076217355937844 [11909.139037] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11909.151954] LustreError: 405556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11909.543941] Lustre: server umount lustre-MDT0001 complete [11923.318519] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11926.411601] Key type lgssc unregistered [11926.736512] LNet: 411759:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11926.745988] LNetError: 411759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11926.757801] LNet: Removed LNI 192.168.203.107@tcp [11927.786267] LNet: Removed LNI 192.168.203.107@tcp1 [11928.862785] Key type .llcrypt unregistered [11928.868698] Key type ._llcrypt unregistered [11949.935262] Key type ._llcrypt registered [11949.938487] Key type .llcrypt registered [11950.072908] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [11950.405365] Key type .llcrypt unregistered [11950.409939] Key type ._llcrypt unregistered [11966.971704] Key type ._llcrypt registered [11966.974754] Key type .llcrypt registered [11967.086753] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [11968.231477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11968.306153] alg: No test for adler32 (adler32-zlib) [11969.389468] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11969.580366] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [11971.231178] Key type lgssc registered [11972.282770] Lustre: Echo OBD driver; http://www.lustre.org/ [11993.855860] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [11995.264455] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [12006.050835] loop0: detected capacity change from 0 to 204800000 [12006.262204] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12006.270045] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12006.303308] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12023.702865] loop0: detected capacity change from 0 to 204800000 [12023.924158] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12023.934982] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12023.958401] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12042.328309] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12042.351720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.615448] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12043.639620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12043.790947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12047.865654] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12057.385742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12057.608774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12057.676084] Lustre: 415593:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12058.052563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12064.347868] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12074.519552] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12081.763247] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12086.718734] loop0: detected capacity change from 0 to 204800000 [12105.065923] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12105.383134] Lustre: sv17746-OST0000: new disk, initializing [12105.392163] Lustre: srv-sv17746-OST0000: No data found on store. Initialize space. [12105.404433] Lustre: 417668:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12105.472198] Lustre: sv17746-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12111.371316] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12120.687135] loop1: detected capacity change from 0 to 204800000 [12140.658041] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12140.915293] Lustre: sv17746-OST0001: new disk, initializing [12140.920529] Lustre: srv-sv17746-OST0001: No data found on store. Initialize space. [12140.930531] Lustre: 418786:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12141.074436] Lustre: sv17746-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12147.466487] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12165.871188] Lustre: 419873:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12165.887181] Lustre: 419873:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12165.898352] Lustre: 419873:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12165.914607] Lustre: 419873:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12165.926671] Lustre: 419873:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12165.935842] Lustre: 419873:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12167.018350] Lustre: 419921:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12167.021990] Lustre: 419921:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 255 previous similar messages [12167.026084] Lustre: 419921:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12167.029636] Lustre: 419921:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 255 previous similar messages [12167.033286] Lustre: 419921:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12167.038709] Lustre: 419921:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 255 previous similar messages [12167.042848] Lustre: 419921:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12167.046843] Lustre: 419921:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 255 previous similar messages [12167.050728] Lustre: 419921:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12167.054578] Lustre: 419921:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 255 previous similar messages [12167.057654] Lustre: 419921:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12167.061492] Lustre: 419921:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 255 previous similar messages [12169.021713] Lustre: server umount sv17746-OST0001 complete [12179.560661] Lustre: server umount sv17746-OST0000 complete [12191.202178] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12191.212881] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12191.232695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12195.299819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12195.300807] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12200.416622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12200.421954] Lustre: Skipped 2 previous similar messages [12205.023595] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12205.240409] Lustre: server umount lustre-MDT0000 complete [12209.360620] LustreError: 415584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165218 with bad export cookie 16716855903964436814 [12209.362843] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12209.377139] LustreError: 415584:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12210.110491] Lustre: server umount lustre-MDT0001 complete [12225.542359] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [12229.144540] Key type lgssc unregistered [12229.660863] LNet: 421932:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12229.683336] LNetError: 421932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12229.703805] LNet: Removed LNI 192.168.203.107@tcp [12230.758384] LNet: Removed LNI 192.168.203.107@tcp1 [12232.579720] Key type .llcrypt unregistered [12232.581496] Key type ._llcrypt unregistered [12255.917147] Key type ._llcrypt registered [12255.919514] Key type .llcrypt registered [12256.062348] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [12256.384886] Key type .llcrypt unregistered [12256.394677] Key type ._llcrypt unregistered [12273.380222] Key type ._llcrypt registered [12273.382609] Key type .llcrypt registered [12273.471915] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [12274.234219] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12274.340575] alg: No test for adler32 (adler32-zlib) [12275.440664] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12275.658910] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [12277.311766] Key type lgssc registered [12278.344169] Lustre: Echo OBD driver; http://www.lustre.org/ [12303.457504] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [12304.922631] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [12315.015704] loop0: detected capacity change from 0 to 204800000 [12315.132210] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12315.146051] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12315.160991] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12333.073366] loop0: detected capacity change from 0 to 204800000 [12333.271775] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12333.296280] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12333.314094] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12351.543945] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12351.559149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12352.854294] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12352.891106] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12353.100840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12357.621958] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12367.719944] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12367.913175] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12367.979920] Lustre: 425757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12368.421357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12373.075318] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12381.621543] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12388.128654] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12394.869830] loop0: detected capacity change from 0 to 204800000 [12414.919731] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12415.209820] Lustre: sv12632-OST0000: new disk, initializing [12415.213311] Lustre: srv-sv12632-OST0000: No data found on store. Initialize space. [12415.219263] Lustre: 427838:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12415.276304] Lustre: sv12632-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12423.128505] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12437.332530] Lustre: 428779:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12437.338522] Lustre: 428779:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12437.343888] Lustre: 428779:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12437.351062] Lustre: 428779:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12437.360235] Lustre: 428779:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12437.367170] Lustre: 428779:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12443.101118] Lustre: server umount sv12632-OST0000 complete [12455.400956] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12455.407580] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12455.428056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12458.976376] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12458.977282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12459.002919] Lustre: Skipped 1 previous similar message [12460.974340] Lustre: server umount lustre-MDT0000 complete [12464.856694] LustreError: 425748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165473 with bad export cookie 7157935348026971163 [12464.861601] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12464.876377] LustreError: 425748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12465.168438] Lustre: server umount lustre-MDT0001 complete [12480.845076] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [12484.542284] Key type lgssc unregistered [12485.907973] LNet: 430484:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12485.931532] LNetError: 430484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12485.944247] LNet: Removed LNI 192.168.203.107@tcp [12487.025462] LNet: Removed LNI 192.168.203.107@tcp1 [12487.941576] Key type .llcrypt unregistered [12487.943521] Key type ._llcrypt unregistered [12508.544633] Key type ._llcrypt registered [12508.547267] Key type .llcrypt registered [12508.635876] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [12508.879468] Key type .llcrypt unregistered [12508.883949] Key type ._llcrypt unregistered [12524.666389] Key type ._llcrypt registered [12524.673531] Key type .llcrypt registered [12524.808317] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [12525.670715] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12525.723776] alg: No test for adler32 (adler32-zlib) [12526.813935] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12527.015960] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [12528.671151] Key type lgssc registered [12529.629619] Lustre: Echo OBD driver; http://www.lustre.org/ [12549.727117] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [12550.750039] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [12552.348566] LNet: Added LNI 192.168.203.107@tcp2 [8/256/0/180] [12562.664359] loop0: detected capacity change from 0 to 204800000 [12562.832270] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12562.839033] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12562.853309] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12579.793956] loop0: detected capacity change from 0 to 204800000 [12579.965046] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12579.995818] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12580.022865] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12598.125479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12598.149155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12599.509536] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12599.541756] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12599.816407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12604.891960] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12615.876757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12616.056926] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12616.144601] Lustre: 434404:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12616.682743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12620.681965] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12628.329219] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12634.602867] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12639.899523] loop0: detected capacity change from 0 to 204800000 [12658.796498] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12659.057701] Lustre: sv8077-OST0000: new disk, initializing [12659.061994] Lustre: srv-sv8077-OST0000: No data found on store. Initialize space. [12659.070418] Lustre: 436481:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12659.153616] Lustre: sv8077-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12665.592622] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12675.416055] loop1: detected capacity change from 0 to 204800000 [12694.377915] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12694.534956] Lustre: sv8077-OST0001: new disk, initializing [12694.540593] Lustre: srv-sv8077-OST0001: No data found on store. Initialize space. [12694.556457] Lustre: 437599:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12694.627725] Lustre: sv8077-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12702.584479] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12710.959864] Lustre: DEBUG MARKER: SKIP: conf-sanity test_57p need a remote client with no Lustre server role for routing [12712.163599] Lustre: 438347:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12712.181206] Lustre: 438347:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12712.193361] Lustre: 438347:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12712.203375] Lustre: 438347:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12712.215526] Lustre: 438347:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12712.223529] Lustre: 438347:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12714.088492] Lustre: server umount sv8077-OST0001 complete [12723.191610] Lustre: server umount sv8077-OST0000 complete [12735.968605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12735.977198] Lustre: Skipped 1 previous similar message [12735.979911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12735.985459] Lustre: Skipped 1 previous similar message [12741.088853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12741.095513] Lustre: Skipped 1 previous similar message [12741.988641] Lustre: server umount lustre-MDT0000 complete [12746.345017] LustreError: 434395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165755 with bad export cookie 4602160249279559373 [12746.347549] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12746.359095] LustreError: 434395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12746.795803] Lustre: server umount lustre-MDT0001 complete [12758.748363] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [12761.253900] Key type lgssc unregistered [12761.521305] LNet: 440355:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12761.535157] LNetError: 440355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12761.546687] LNet: Removed LNI 192.168.203.107@tcp [12762.592168] LNet: Removed LNI 192.168.203.107@tcp1 [12762.599242] LNetError: 440355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12762.609090] LNetError: 440355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12762.663555] LNet: Removed LNI 192.168.203.107@tcp2 [12763.576349] Key type .llcrypt unregistered [12763.580742] Key type ._llcrypt unregistered [12784.510146] Key type ._llcrypt registered [12784.512592] Key type .llcrypt registered [12784.650524] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [12784.998290] Key type .llcrypt unregistered [12784.999952] Key type ._llcrypt unregistered [12801.313839] Key type ._llcrypt registered [12801.317257] Key type .llcrypt registered [12801.428697] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [12802.727935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12802.771854] alg: No test for adler32 (adler32-zlib) [12803.959451] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12804.297953] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [12805.983171] Key type lgssc registered [12807.163896] Lustre: Echo OBD driver; http://www.lustre.org/ [12827.282483] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [12828.441697] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [12838.899510] loop0: detected capacity change from 0 to 204800000 [12839.028832] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12839.051635] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12839.065574] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12856.570524] loop0: detected capacity change from 0 to 204800000 [12856.705382] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12856.719668] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12856.741267] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12875.624978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12875.660384] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12877.025925] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12877.090912] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12877.336530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12881.972119] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12891.545673] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12891.674350] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12891.746740] Lustre: 444227:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12892.045475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12896.943430] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12905.674567] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12911.469363] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12917.027718] loop0: detected capacity change from 0 to 204800000 [12936.447256] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12936.823236] Lustre: sv17544-OST0000: new disk, initializing [12936.828833] Lustre: srv-sv17544-OST0000: No data found on store. Initialize space. [12936.839572] Lustre: 446302:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12936.912414] Lustre: sv17544-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12945.092915] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12955.711486] loop1: detected capacity change from 0 to 204800000 [12974.442526] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12974.704197] Lustre: sv17544-OST0001: new disk, initializing [12974.708266] Lustre: srv-sv17544-OST0001: No data found on store. Initialize space. [12974.723569] Lustre: 447424:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12974.811709] Lustre: sv17544-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12982.484456] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [12995.230477] LustreError: 448316:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 sleeping for 15000ms [13010.247231] LustreError: 448316:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 awake [13018.886945] Lustre: 448609:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13018.900524] Lustre: 448609:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13018.912583] Lustre: 448609:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13018.932236] Lustre: 448609:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13018.938448] Lustre: 448609:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13018.949752] Lustre: 448609:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13020.885136] Lustre: server umount sv17544-OST0001 complete [13029.593974] Lustre: server umount sv17544-OST0000 complete [13044.192773] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13044.231708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13044.242670] Lustre: Skipped 1 previous similar message [13049.314635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13049.323938] Lustre: Skipped 1 previous similar message [13049.747543] Lustre: server umount lustre-MDT0000 complete [13053.979862] LustreError: 444219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166062 with bad export cookie 7809265296419337523 [13053.983214] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13053.999617] LustreError: 444219:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13054.347640] Lustre: server umount lustre-MDT0001 complete [13069.435822] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [13073.392808] Key type lgssc unregistered [13073.811647] LNet: 450617:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13073.823151] LNetError: 450617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13073.845662] LNet: Removed LNI 192.168.203.107@tcp [13074.918291] LNet: Removed LNI 192.168.203.107@tcp1 [13076.032269] Key type .llcrypt unregistered [13076.035209] Key type ._llcrypt unregistered [13097.505804] Key type ._llcrypt registered [13097.509244] Key type .llcrypt registered [13097.662262] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [13098.011929] Key type .llcrypt unregistered [13098.017015] Key type ._llcrypt unregistered [13115.398340] Key type ._llcrypt registered [13115.399779] Key type .llcrypt registered [13115.527366] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [13116.907326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13116.990988] alg: No test for adler32 (adler32-zlib) [13118.324794] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13118.615262] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [13120.351614] Key type lgssc registered [13122.041424] Lustre: Echo OBD driver; http://www.lustre.org/ [13144.853041] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [13146.195502] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [13157.612319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13157.624824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13158.942986] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13158.993051] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13159.161612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13164.307469] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13173.807817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13173.894853] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13173.924440] Lustre: 453889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13174.143717] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13178.789593] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13188.537974] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13194.708925] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13199.922269] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [13200.490760] Lustre: DEBUG MARKER: rpc test_57r: @@@@@@ FAIL: Can't determine interface name for NID 192.168.203.107@tcp1 [13202.823958] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing check_logdir /tmp/test_logs/1788166207 [13213.665095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13213.682966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13213.690073] Lustre: Skipped 1 previous similar message [13219.474552] Lustre: server umount lustre-MDT0000 complete [13223.905631] LustreError: 453898:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13223.930570] LustreError: 453898:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [13223.954622] LustreError: 453881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166232 with bad export cookie 574997609062017512 [13223.964327] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13223.984893] LustreError: 453881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13224.019901] LustreError: 453891:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [13224.518415] Lustre: server umount lustre-MDT0001 complete [13244.422507] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [13247.538030] Key type lgssc unregistered [13247.905491] LNet: 457231:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13247.923541] LNetError: 457231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13247.944463] LNet: Removed LNI 192.168.203.107@tcp [13248.997625] LNet: Removed LNI 192.168.203.107@tcp1 [13249.848797] Key type .llcrypt unregistered [13249.850677] Key type ._llcrypt unregistered [13268.569885] Key type ._llcrypt registered [13268.573392] Key type .llcrypt registered [13268.733175] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [13269.921793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13270.047197] alg: No test for adler32 (adler32-zlib) [13271.184167] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13271.415560] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [13273.104194] Key type lgssc registered [13274.305026] Lustre: Echo OBD driver; http://www.lustre.org/ [13281.159317] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [13282.187567] LNet: Added LNI 192.168.203.107@tcp1 [8/256/0/180] [13289.535812] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13289.573392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13290.991241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13295.788536] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13301.216695] LustreError: 458715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13301.241192] LustreError: 458715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13306.125859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13306.343783] LustreError: 458714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13306.761106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13312.217646] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13321.166681] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13327.278306] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13335.455516] LNet: Removed LNI 192.168.203.107@tcp1 [13336.259485] Lustre: 460426:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13336.290894] Lustre: 460426:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13336.311373] Lustre: 460426:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13336.327949] Lustre: 460426:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13336.343386] Lustre: 460426:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13336.353974] Lustre: 460426:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13342.096849] Lustre: server umount lustre-MDT0000 complete [13342.689400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13342.701798] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13342.706442] LustreError: 458715:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13342.712282] Lustre: Skipped 1 previous similar message [13342.761586] LustreError: 458715:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13347.034500] LustreError: 458694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166355 with bad export cookie 8374901585313983413 [13347.050922] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13347.052701] LustreError: 458694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13347.083252] Lustre: 458706:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.203.107@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [13349.620482] Lustre: server umount lustre-MDT0001 complete [13362.570810] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [13365.674331] Key type lgssc unregistered [13365.986507] LNet: 461432:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13365.997380] LNetError: 461432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13367.022418] LNet: Removed LNI 192.168.203.107@tcp [13368.279686] Key type .llcrypt unregistered [13368.284723] Key type ._llcrypt unregistered [13393.168928] Key type ._llcrypt registered [13393.170926] Key type .llcrypt registered [13393.309729] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [13394.280215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13394.349703] alg: No test for adler32 (adler32-zlib) [13395.459207] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13395.662174] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [13397.287519] Key type lgssc registered [13398.584449] Lustre: Echo OBD driver; http://www.lustre.org/ [13407.515674] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13407.552793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13408.978057] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13409.022257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13414.152509] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13419.493771] LustreError: 462753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13419.512202] LustreError: 462753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13423.565099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13423.883520] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13423.895849] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13423.998273] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13429.027489] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13437.300296] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13442.877416] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13452.223590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13452.451163] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13452.593727] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13452.607291] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13452.647736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13458.989556] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13460.996396] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13461.021145] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13467.747348] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13480.423233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13480.430317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13480.441457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13481.953660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13481.956413] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13481.958606] Lustre: Skipped 1 previous similar message [13484.368938] Lustre: server umount lustre-MDT0000 complete [13486.021424] LustreError: 462748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13487.078598] LustreError: 463488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13490.027744] LustreError: 462733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166498 with bad export cookie 5506791028873839662 [13490.028830] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13490.046158] LustreError: 462733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13491.129317] LustreError: 462748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13491.185476] LustreError: 462748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13491.200541] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [13491.243961] Lustre: Skipped 2 previous similar messages [13492.193718] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13492.228243] Lustre: Skipped 1 previous similar message [13495.200893] LustreError: 465259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13495.201674] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13495.245723] LustreError: 465259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13495.255521] Lustre: Skipped 1 previous similar message [13496.356864] Lustre: server umount lustre-MDT0001 complete [13505.917311] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [13515.129662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13535.202942] LustreError: 462062:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001f9ac4c9 x1875028375276288/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13535.584093] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13535.596294] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [13535.612426] LustreError: 464583:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13535.635360] LustreError: 464583:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13535.706723] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13535.788351] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [13540.713303] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13549.063976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13549.374889] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13549.384978] LNetError: 462061:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [13549.568058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13550.579806] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13554.076124] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13561.958562] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13563.436462] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13569.590794] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13571.046144] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [13576.674314] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13576.682902] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13576.699284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13580.255996] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13582.679849] Lustre: server umount lustre-OST0000 complete [13586.404155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13586.405449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13586.427690] Lustre: Skipped 2 previous similar messages [13586.450109] Lustre: Skipped 4 previous similar messages [13592.452796] Lustre: server umount lustre-MDT0000 complete [13596.355041] LustreError: 465552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166605 with bad export cookie 5506791028873846928 [13596.360735] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13596.367819] LustreError: 465552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13596.682703] Lustre: server umount lustre-MDT0001 complete [13608.595419] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [13611.701498] Key type lgssc unregistered [13612.135238] LNet: 469036:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13612.142315] LNetError: 469036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13613.160830] LNet: Removed LNI 192.168.203.107@tcp [13614.069210] Key type .llcrypt unregistered [13614.072234] Key type ._llcrypt unregistered [13641.657571] Key type ._llcrypt registered [13641.659808] Key type .llcrypt registered [13641.776179] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [13643.164996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13643.226969] alg: No test for adler32 (adler32-zlib) [13644.289373] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13644.553458] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [13646.247210] Key type lgssc registered [13647.196705] Lustre: Echo OBD driver; http://www.lustre.org/ [13654.122610] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13654.149010] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13655.462937] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13655.507272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13659.534359] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13665.774167] LustreError: 470457:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13665.788936] LustreError: 470457:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [13667.440239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13667.683408] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13667.693486] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 4 previous similar messages [13667.829807] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13671.952417] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13679.692339] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13685.562380] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13691.361861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13691.368334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13691.378251] Lustre: Skipped 1 previous similar message [13692.459062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13692.469204] Lustre: Skipped 2 previous similar messages [13694.806890] Lustre: server umount lustre-MDT0000 complete [13697.505920] LustreError: 470456:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13697.533927] LustreError: 470456:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13698.134974] LustreError: 470436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166706 with bad export cookie 11340626471430764405 [13698.145910] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13698.485030] Lustre: server umount lustre-MDT0001 complete [13712.297586] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13712.519938] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13712.540900] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 5 previous similar messages [13712.647182] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13712.678939] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13712.830792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13716.909794] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13724.702143] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13724.811741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13724.861317] Lustre: 473045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13725.014336] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13725.021282] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [13725.105322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13729.442978] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13731.375238] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13731.386277] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [13736.358589] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13741.629687] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13750.516617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13750.824778] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [13750.837854] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [13750.853099] LustreError: 474929:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [13750.860574] LustreError: 474929:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [13750.866866] LustreError: 474929:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [13750.982982] Lustre: server umount lustre-OST0000 complete [13750.988915] LustreError: 474929:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [13761.840090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13762.056874] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13762.226504] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13762.275043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13768.311370] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13772.437779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [13774.841273] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13782.832324] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13782.945583] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [13783.012244] Lustre: lustre-OST0001: new disk, initializing [13783.018535] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13783.035516] Lustre: 476469:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13783.106407] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.203.107@tcp1 has no available nets [13783.116790] LNetError: 469764:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 6 previous similar messages [13783.153118] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13788.632443] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13789.778853] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13789.787984] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13789.822594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13795.370808] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13800.416084] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13800.418399] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13800.424027] LustreError: Skipped 1 previous similar message [13800.436490] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13800.439651] Lustre: Skipped 1 previous similar message [13800.461856] Lustre: Skipped 1 previous similar message [13804.341278] Lustre: server umount lustre-OST0001 complete [13805.549994] LustreError: 475397:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13805.576049] LustreError: 475397:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13808.098483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13808.105696] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13808.119952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13810.663956] LustreError: 476902:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13810.676946] LustreError: 476902:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13814.169627] Lustre: server umount lustre-OST0000 complete [13820.897176] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13820.922782] Lustre: Skipped 1 previous similar message [13820.932798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13820.937945] Lustre: Skipped 3 previous similar messages [13824.529761] Lustre: server umount lustre-MDT0000 complete [13826.016533] LustreError: 474159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13826.034287] LustreError: 474159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13828.501137] LustreError: 475412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166837 with bad export cookie 11340626471430764958 [13828.512698] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13828.525050] LustreError: 475412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [13828.956971] Lustre: server umount lustre-MDT0001 complete [13839.308178] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [13841.664570] Key type lgssc unregistered [13841.902599] LNet: 478427:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13841.908187] LNetError: 478427:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13841.923139] LNet: Removed LNI 192.168.203.107@tcp [13842.614891] Key type .llcrypt unregistered [13842.616937] Key type ._llcrypt unregistered [13914.364241] Key type ._llcrypt registered [13914.368190] Key type .llcrypt registered [13914.479724] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [13936.865599] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [13939.128265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13939.165091] alg: No test for adler32 (adler32-zlib) [13940.149450] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13940.300459] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [13941.935195] Key type lgssc registered [13943.064963] Lustre: Echo OBD driver; http://www.lustre.org/ [13976.925667] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13976.937271] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13978.185036] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13978.228618] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13978.321828] Lustre: lustre-MDT0000: new disk, initializing [13978.421335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13978.444679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13982.282723] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [13991.665686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13991.821627] Lustre: 483375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13991.899834] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13991.903446] Lustre: Skipped 1 previous similar message [13991.969274] Lustre: lustre-MDT0001: new disk, initializing [13992.072558] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13992.101746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13992.115924] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13995.683279] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14004.076735] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14009.235473] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14015.037226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14015.267436] Lustre: lustre-OST0000: new disk, initializing [14015.272358] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14015.279604] Lustre: 485341:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14015.348324] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14021.268992] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14021.732114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14021.743171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14021.831994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14029.732287] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14033.332725] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14033.504815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14036.979868] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14037.108558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14042.598230] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14042.601180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14042.604385] LustreError: Skipped 1 previous similar message [14042.605559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14042.615029] Lustre: Skipped 1 previous similar message [14042.631567] Lustre: Skipped 1 previous similar message [14044.666379] Lustre: server umount lustre-OST0000 complete [14048.225374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14048.231253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14048.239993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14052.835143] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14052.835713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14052.878107] Lustre: Skipped 1 previous similar message [14054.399631] Lustre: server umount lustre-MDT0000 complete [14058.134171] LustreError: 483365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167066 with bad export cookie 7369956095035548037 [14058.149896] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14058.150954] LustreError: 483365:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14058.637230] Lustre: server umount lustre-MDT0001 complete [14068.022842] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:04:35 (1788167075) [14074.941832] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:04:42 (1788167082) [14082.463199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14082.998092] LustreError: 488097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14083.017843] LustreError: 488097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14083.084511] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14086.444135] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14088.161216] LustreError: 488098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14088.174333] LustreError: 488098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14093.282941] LustreError: 488097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14093.633859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14093.883616] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14098.190812] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14104.998326] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14110.032615] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14115.806254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14116.076381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14120.910707] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14127.211202] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14133.707683] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [14135.263286] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [14136.675925] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [14138.374155] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [14140.132639] Lustre: Failing over lustre-MDT0000 [14140.417215] Lustre: server umount lustre-MDT0000 complete [14142.949959] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14142.961224] Lustre: Skipped 2 previous similar messages [14142.961427] LustreError: 488094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14142.995850] LustreError: 488094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14148.067055] LustreError: 489121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14148.098764] LustreError: 489121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14158.303177] Lustre: 481597:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788167151/real 1788167151] req@00000000f6072bf4 x1875028946780800/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788167167 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14158.313116] LustreError: 489121:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14158.350408] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14158.370114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14158.388684] LustreError: 489121:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14168.786941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14168.844369] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14170.046110] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14172.611580] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14174.189583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [14174.224020] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [14174.283177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [14181.135578] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14182.652245] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14185.852874] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [14187.759305] Lustre: Failing over lustre-MDT0000 [14188.002558] Lustre: server umount lustre-MDT0000 complete [14189.550369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14189.553020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14189.560516] LustreError: 489122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14189.570862] Lustre: Skipped 2 previous similar messages [14189.595653] LustreError: 489122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [14205.919170] Lustre: 481597:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788167198/real 1788167198] req@00000000235b9925 x1875028946800512/t0(0) o400->MGC192.168.203.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788167214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14205.962600] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14206.196986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14216.164059] LustreError: 481594:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000051d39e04 x1875028946806528/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14216.427603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14216.477764] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [14220.734914] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14221.250861] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [14221.810564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [14221.819399] Lustre: Skipped 2 previous similar messages [14221.843942] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [14221.889832] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [14229.647928] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14230.987738] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14237.154094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14237.156172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14237.156990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14237.177035] Lustre: Skipped 2 previous similar messages [14237.203034] Lustre: Skipped 2 previous similar messages [14242.274277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14242.592203] Lustre: server umount lustre-MDT0000 complete [14246.327044] LustreError: 488077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167255 with bad export cookie 7369956095035550760 [14246.331759] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14246.336041] LustreError: 488077:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14246.688935] Lustre: server umount lustre-MDT0001 complete [14259.671417] Lustre: server umount lustre-OST0000 complete [14273.380382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14273.796076] LustreError: 494442:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14273.816626] LustreError: 494442:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 24 previous similar messages [14277.831472] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14286.099801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14286.396242] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14286.402688] Lustre: Skipped 1 previous similar message [14291.018451] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14297.949442] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14303.874541] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14311.162828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14317.818450] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14318.661065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [14325.156269] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14331.982552] Lustre: server umount lustre-OST0000 complete [14337.506801] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14337.515932] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14337.531199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14337.536853] Lustre: Skipped 2 previous similar messages [14346.281849] Lustre: server umount lustre-MDT0000 complete [14349.216546] Lustre: 481598:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788167341/real 1788167341] req@000000007d5aaa87 x1875028946850432/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788167357 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14349.261907] LustreError: 495171:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14349.283832] LustreError: 495171:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14349.933277] LustreError: 494424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167358 with bad export cookie 7369956095035551495 [14349.946668] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14349.946720] LustreError: 494424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14350.259122] Lustre: server umount lustre-MDT0001 complete [14360.501338] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [14363.117156] Key type lgssc unregistered [14363.394868] LNet: 497972:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14363.406086] LNetError: 497972:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14363.428039] LNet: Removed LNI 192.168.203.107@tcp [14364.339096] Key type .llcrypt unregistered [14364.340766] Key type ._llcrypt unregistered [14392.323925] Key type ._llcrypt registered [14392.329686] Key type .llcrypt registered [14392.415551] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [14393.899647] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14393.906730] alg: No test for adler32 (adler32-zlib) [14395.053453] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14395.293604] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [14396.969621] Key type lgssc registered [14398.499580] Lustre: Echo OBD driver; http://www.lustre.org/ [14406.539307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14406.572525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14408.039534] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14411.851578] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14417.383803] LustreError: 499492:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14417.400322] LustreError: 499492:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14419.607078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14419.935241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14424.066415] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14431.697667] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14438.185489] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14445.331751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14445.756893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14452.239137] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14452.942460] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14460.547917] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14475.751157] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14475.759183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14475.767398] Lustre: Skipped 2 previous similar messages [14475.781079] Lustre: Skipped 2 previous similar messages [14477.076286] Lustre: server umount lustre-MDT0000 complete [14480.870968] LustreError: 500535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14480.883364] LustreError: 500535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14480.889257] LustreError: 501996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167489 with bad export cookie 8976454543480517998 [14480.894502] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14480.906811] LustreError: 501996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14481.173440] Lustre: server umount lustre-MDT0001 complete [14495.987846] Lustre: server umount lustre-OST0000 complete [14509.195247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14509.638316] LustreError: 503225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14509.705385] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14514.013399] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14515.170981] LustreError: 503226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14520.293621] LustreError: 503225:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14522.352221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14522.741282] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14527.053839] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14533.821625] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14539.248623] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14545.694477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14546.194832] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14552.351609] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14555.446970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [14558.480453] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14570.983130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14570.984407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14571.004155] Lustre: Skipped 1 previous similar message [14571.016897] Lustre: Skipped 2 previous similar messages [14576.101636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14576.116710] Lustre: Skipped 2 previous similar messages [14576.391658] Lustre: server umount lustre-MDT0000 complete [14580.044551] LustreError: 503206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167588 with bad export cookie 8976454543480519111 [14580.059561] LustreError: 503206:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14580.064486] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14580.319655] Lustre: server umount lustre-MDT0001 complete [14594.178764] Lustre: server umount lustre-OST0000 complete [14610.220622] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14610.770108] LustreError: 507136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14610.883467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14615.254766] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14624.077908] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14628.452680] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14635.673596] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14642.676612] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14650.630084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14651.274064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14651.287174] Lustre: Skipped 1 previous similar message [14655.355640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [14659.457988] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14667.928085] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14675.939036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14675.942818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14675.943133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14675.957576] Lustre: Skipped 2 previous similar messages [14678.584794] Lustre: server umount lustre-OST0000 complete [14685.664418] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14685.678529] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14685.690177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14685.694346] Lustre: Skipped 1 previous similar message [14688.386852] Lustre: server umount lustre-MDT0000 complete [14691.300307] LustreError: 507136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14691.313105] LustreError: 507136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14691.353216] LustreError: 507118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167700 with bad export cookie 8976454543480520133 [14691.363071] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14691.363926] LustreError: 507118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14691.588191] Lustre: server umount lustre-MDT0001 complete [14704.181402] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [14707.289250] Key type lgssc unregistered [14707.507787] LNet: 510667:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14707.514060] LNetError: 510667:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14707.553415] LNet: Removed LNI 192.168.203.107@tcp [14708.539855] Key type .llcrypt unregistered [14708.543413] Key type ._llcrypt unregistered [14735.351507] Key type ._llcrypt registered [14735.353511] Key type .llcrypt registered [14735.500218] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [14736.770746] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14736.947236] alg: No test for adler32 (adler32-zlib) [14738.059969] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14738.304166] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [14739.983210] Key type lgssc registered [14741.202854] Lustre: Echo OBD driver; http://www.lustre.org/ [14749.348914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14749.389876] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14749.400518] LustreError: 512109:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [14749.422308] LustreError: 512109:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [14749.426134] LustreError: 512109:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [14749.434606] LustreError: 512109:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [14749.444644] LustreError: 512109:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14756.862252] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [14756.871285] LustreError: 512506:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [14756.890129] LustreError: 512506:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [14756.896131] LustreError: 512506:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [14756.903242] LustreError: 512506:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [14756.922154] LustreError: 512506:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14779.335964] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [14782.373277] Key type lgssc unregistered [14782.659519] LNet: 513510:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14782.663775] LNetError: 513510:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14782.676814] LNet: Removed LNI 192.168.203.107@tcp [14783.382920] Key type .llcrypt unregistered [14783.387855] Key type ._llcrypt unregistered [14804.091989] Key type ._llcrypt registered [14804.094506] Key type .llcrypt registered [14804.182541] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [14819.107341] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [14820.075997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14820.251869] alg: No test for adler32 (adler32-zlib) [14821.341767] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14821.544881] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [14823.191130] Key type lgssc registered [14824.285029] Lustre: Echo OBD driver; http://www.lustre.org/ [14860.341710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14860.371319] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14861.686296] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14861.732901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14861.820247] Lustre: lustre-MDT0000: new disk, initializing [14861.903379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14861.932419] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14866.173270] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14875.765049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14875.878514] Lustre: 516666:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14875.927444] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14875.931169] Lustre: Skipped 1 previous similar message [14876.001192] Lustre: lustre-MDT0001: new disk, initializing [14876.099420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14876.135516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14876.141484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14880.103389] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14888.633851] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14893.628572] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14899.761960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14900.018823] Lustre: lustre-OST0000: new disk, initializing [14900.022789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14900.030900] Lustre: 518635:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14900.099854] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14905.526544] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14907.447783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14907.456625] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14907.532424] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14914.580791] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14919.646395] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14919.831741] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14924.149894] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14924.348361] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14928.356318] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14928.356773] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14928.361697] LustreError: Skipped 1 previous similar message [14928.377117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14932.394291] Lustre: server umount lustre-OST0000 complete [14937.569769] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14937.579961] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14937.588509] Lustre: Skipped 1 previous similar message [14937.594298] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14937.599678] Lustre: Skipped 1 previous similar message [14938.594198] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14938.633397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14938.644719] Lustre: Skipped 1 previous similar message [14943.715144] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14943.719077] Lustre: Skipped 1 previous similar message [14948.836242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14948.864458] Lustre: Skipped 1 previous similar message [14950.367320] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14950.672169] Lustre: server umount lustre-MDT0000 complete [14955.501539] LustreError: 518634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167964 with bad export cookie 6676781362615685694 [14955.504123] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14955.530836] LustreError: 518634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14955.856939] Lustre: server umount lustre-MDT0001 complete [14964.924224] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [14966.419506] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:19:34 (1788167974) [14972.375852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14972.733840] LustreError: 521093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14972.751029] LustreError: 521093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14972.804435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14976.666909] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14978.021200] LustreError: 521094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14983.140756] LustreError: 521093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14984.707936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14985.048745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14989.415244] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [14996.576691] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15002.309478] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15008.928140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15009.223863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15015.826456] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15023.845799] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15031.646888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15031.825127] Lustre: lustre-OST0001: new disk, initializing [15031.832966] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [15031.841813] Lustre: 523931:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15031.950047] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15038.284581] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15039.140589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15039.146150] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15039.263932] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15047.609323] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15054.297961] Lustre: lustre-OST0001: Not available for connect from 192.168.203.7@tcp (stopping) [15054.820649] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15054.821227] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15054.841100] LustreError: Skipped 1 previous similar message [15054.857025] Lustre: Skipped 1 previous similar message [15057.044672] Lustre: server umount lustre-OST0001 complete [15059.385992] LustreError: 523367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15065.080776] LustreError: 522914:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15065.118879] LustreError: 522914:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15075.302447] LustreError: 524853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15075.306660] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15075.324460] LustreError: 524853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15075.350155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15075.362672] Lustre: Skipped 3 previous similar messages [15080.255619] Lustre: server umount lustre-OST0000 complete [15085.541950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15085.558239] Lustre: Skipped 2 previous similar messages [15090.424658] Lustre: server umount lustre-MDT0000 complete [15094.002217] LustreError: 523930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168102 with bad export cookie 6676781362615686576 [15094.006868] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15094.012852] LustreError: 523930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15100.368730] Lustre: server umount lustre-MDT0001 complete [15111.417992] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [15114.179695] Key type lgssc unregistered [15114.432900] LNet: 525916:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15114.444077] LNetError: 525916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15114.462580] LNet: Removed LNI 192.168.203.107@tcp [15115.219500] Key type .llcrypt unregistered [15115.222419] Key type ._llcrypt unregistered [15169.924383] Key type ._llcrypt registered [15169.928233] Key type .llcrypt registered [15170.045973] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [15171.360388] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15171.388439] alg: No test for adler32 (adler32-zlib) [15172.483906] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15172.736226] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [15174.463157] Key type lgssc registered [15175.702604] Lustre: Echo OBD driver; http://www.lustre.org/ [15183.794319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15183.825892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15185.043087] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15185.059254] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15185.208475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15190.054168] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15199.115686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15199.186443] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15199.225650] Lustre: 528716:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15199.526475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15204.221880] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15211.652671] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15216.796290] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15220.196091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15220.199782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15220.213745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15223.266619] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15223.267034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15223.281968] Lustre: Skipped 1 previous similar message [15224.946888] Lustre: server umount lustre-MDT0000 complete [15228.185316] LustreError: 528706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168236 with bad export cookie 8066826801545576890 [15228.190768] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15228.198483] LustreError: 528706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15228.534863] Lustre: server umount lustre-MDT0001 complete [15238.124819] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:24:06 (1788168246) [15244.567982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15244.963270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15249.412939] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15250.401039] LustreError: 531227:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15250.417067] LustreError: 531227:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15255.525309] LustreError: 531226:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15257.511247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15257.774502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15261.712079] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15269.336298] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15274.941653] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15281.263259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15281.471150] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15281.674720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15287.688658] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15294.627225] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15302.341288] Lustre: Permanently deactivating lustre-OST0000 [15302.362317] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [15308.912413] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15310.328487] LustreError: 534166:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [15310.333762] LustreError: 534166:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15310.337322] LustreError: 534166:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15313.674488] LustreError: 534214:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15313.679602] LustreError: 534214:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15322.035123] Lustre: server umount lustre-OST0000 complete [15325.335561] LustreError: 534464:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15325.343975] LustreError: 534464:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15329.248553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15329.261056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15329.276516] Lustre: Skipped 1 previous similar message [15334.371107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15334.381332] Lustre: Skipped 1 previous similar message [15336.474527] Lustre: server umount lustre-MDT0000 complete [15339.488942] LustreError: 531221:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15339.503740] LustreError: 531221:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15340.426665] LustreError: 533916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168349 with bad export cookie 8066826801545577471 [15340.433844] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15340.440017] LustreError: 533916:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15350.908778] Lustre: server umount lustre-MDT0001 complete [15358.521988] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15363.507541] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15366.183767] LustreError: 535622:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [15372.266498] Lustre: 535961:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.107@tcp [15374.003738] Lustre: server umount MGS complete [15384.697694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15385.259363] LustreError: 536685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15385.292549] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15385.297900] Lustre: Skipped 1 previous similar message [15385.377250] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15389.994289] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15390.691368] LustreError: 536686:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15398.469599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15398.748949] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15403.196470] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15410.725875] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15417.082335] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15418.987183] Lustre: Permanently reactivating lustre-OST0000 [15419.022161] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [15419.027633] Lustre: Skipped 2 previous similar messages [15447.519145] LustreError: 538487:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [15453.721800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15453.959150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15453.963291] Lustre: Skipped 1 previous similar message [15458.965867] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15465.936151] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15472.616489] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15472.621560] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15472.633912] Lustre: Skipped 1 previous similar message [15472.648175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15473.641881] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15476.869662] Lustre: server umount lustre-OST0000 complete [15480.803416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15480.809422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15480.826202] Lustre: Skipped 1 previous similar message [15480.833291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15480.840833] Lustre: Skipped 2 previous similar messages [15486.654788] LustreError: 540345:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000a2052c69) refcount nonzero (2) after lock cleanup; forcing cleanup. [15488.709619] Lustre: server umount lustre-MDT0000 complete [15488.996695] LustreError: 536685:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15489.024632] LustreError: 536685:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15492.380565] LustreError: 536667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168501 with bad export cookie 8066826801545578521 [15492.386921] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15492.394107] LustreError: 536667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15492.725790] Lustre: server umount lustre-MDT0001 complete [15503.718061] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [15506.061260] Key type lgssc unregistered [15506.290570] LNet: 541099:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15506.303862] LNetError: 541099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15506.337668] LNet: Removed LNI 192.168.203.107@tcp [15507.134891] Key type .llcrypt unregistered [15507.137268] Key type ._llcrypt unregistered [15523.417345] Key type ._llcrypt registered [15523.419947] Key type .llcrypt registered [15523.570947] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [15538.644122] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [15539.420502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15539.482475] alg: No test for adler32 (adler32-zlib) [15540.661169] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15540.887781] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [15542.520657] Key type lgssc registered [15543.725606] Lustre: Echo OBD driver; http://www.lustre.org/ [15582.392800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15582.415303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15583.750900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15583.778604] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15583.923866] Lustre: lustre-MDT0000: new disk, initializing [15584.040344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15584.078601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15589.191809] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15600.148147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15600.282363] Lustre: 544258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15600.370652] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15600.374499] Lustre: Skipped 1 previous similar message [15600.453920] Lustre: lustre-MDT0001: new disk, initializing [15600.594601] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15600.635771] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15600.644234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15605.101264] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15614.493461] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15620.586370] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15627.284280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15627.534539] Lustre: lustre-OST0000: new disk, initializing [15627.538099] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15627.543524] Lustre: 546227:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15627.615270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15633.515098] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15635.514820] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15635.532585] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15635.624639] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15643.331059] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15647.840763] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15648.052404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15652.591353] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15652.804964] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15656.419524] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15656.421487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15656.441511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15656.457530] LustreError: Skipped 1 previous similar message [15661.552595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15661.562886] Lustre: Skipped 1 previous similar message [15666.666341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15666.677683] Lustre: Skipped 1 previous similar message [15669.216956] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15669.467895] Lustre: server umount lustre-OST0000 complete [15676.898194] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15676.902260] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15676.917111] Lustre: Skipped 2 previous similar messages [15679.763942] Lustre: server umount lustre-MDT0000 complete [15683.313843] LustreError: 544248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168692 with bad export cookie 17240629937195520031 [15683.314747] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15683.321640] LustreError: 544248:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15683.620214] Lustre: server umount lustre-MDT0001 complete [15692.221666] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [15693.887515] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:31:41 (1788168701) [15701.003251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15701.413392] LustreError: 548687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15701.441288] LustreError: 548687:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15701.483914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15705.625692] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15706.594457] LustreError: 548688:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15711.712380] LustreError: 548687:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15712.892798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15713.213610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15716.910925] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15724.629206] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15730.672319] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15737.757199] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15737.831170] LustreError: 550508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15738.091319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15744.275473] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15752.346536] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15754.295269] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [15757.792201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [15763.939293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15763.964682] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15764.000405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15764.007308] Lustre: Skipped 1 previous similar message [15767.017344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15767.024629] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15770.119933] Lustre: server umount lustre-OST0000 complete [15774.688784] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15774.701255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15774.715239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15774.726101] Lustre: Skipped 2 previous similar messages [15780.160993] Lustre: server umount lustre-MDT0000 complete [15782.372205] LustreError: 549928:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15782.400661] LustreError: 549928:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15784.352381] LustreError: 548667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168793 with bad export cookie 17240629937195520913 [15784.363307] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15784.363875] LustreError: 548667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15784.782361] Lustre: server umount lustre-MDT0001 complete [15795.278757] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [15797.865639] Key type lgssc unregistered [15798.139914] LNet: 552409:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15798.150539] LNetError: 552409:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15798.172801] LNet: Removed LNI 192.168.203.107@tcp [15799.013322] Key type .llcrypt unregistered [15799.020961] Key type ._llcrypt unregistered [15822.376776] Key type ._llcrypt registered [15822.378956] Key type .llcrypt registered [15822.471898] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [15822.723418] Key type .llcrypt unregistered [15822.728090] Key type ._llcrypt unregistered [15840.415808] Key type ._llcrypt registered [15840.418684] Key type .llcrypt registered [15840.543222] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [15841.744849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15841.793785] alg: No test for adler32 (adler32-zlib) [15842.820649] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15843.031372] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [15844.735176] Key type lgssc registered [15845.787469] Lustre: Echo OBD driver; http://www.lustre.org/ [15853.265558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15853.296446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15854.837503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15859.920208] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15865.323464] LustreError: 554622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15865.377452] LustreError: 554622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15869.425535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15869.861809] LustreError: 554621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15869.905856] LustreError: 554621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15869.921267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15875.049695] LustreError: 554617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15875.084387] LustreError: 554617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15875.545164] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15880.173230] LustreError: 554618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15880.202487] LustreError: 554618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15881.202424] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [15883.458974] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15886.315926] LustreError: 554617:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15886.332059] LustreError: 554617:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15890.315365] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15890.715213] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15895.306330] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [15905.488494] Lustre: server umount lustre-OST0000 complete [15910.886224] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [15910.897631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15910.913542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15916.514964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15916.541548] Lustre: Skipped 3 previous similar messages [15918.625623] Lustre: server umount lustre-MDT0000 complete [15921.633497] Lustre: 553931:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788168914/real 1788168914] req@000000002a5b074e x1875030941132416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788168930 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15921.641725] LustreError: 554616:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15921.657073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15921.665969] LustreError: 554616:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15921.683938] Lustre: Skipped 1 previous similar message [15921.818199] LustreError: 554602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168930 with bad export cookie 6765938188506086638 [15921.835801] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15922.129454] Lustre: server umount lustre-MDT0001 complete [15931.403876] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [15933.791503] Key type lgssc unregistered [15934.047955] LNet: 557771:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15934.059368] LNetError: 557771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15934.080651] LNet: Removed LNI 192.168.203.107@tcp [15934.799657] Key type .llcrypt unregistered [15934.803380] Key type ._llcrypt unregistered [15957.447822] Key type ._llcrypt registered [15957.450450] Key type .llcrypt registered [15957.566898] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [15958.460957] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15958.475346] alg: No test for adler32 (adler32-zlib) [15959.551374] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15959.769103] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [15961.415166] Key type lgssc registered [15962.280327] Lustre: Echo OBD driver; http://www.lustre.org/ [15969.289069] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15969.331371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15984.863439] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [15989.983524] LustreError: 559159:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.107@tcp: failed processing log, type 4: rc = -110 [16016.672068] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16022.821190] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16030.439674] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16037.859137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16038.301316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16038.355133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16042.793452] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16043.491104] LustreError: 559185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16043.505423] LustreError: 559185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16048.622743] LustreError: 559184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16051.598059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16051.692060] LustreError: 559185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16051.706448] LustreError: 559185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16052.036780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16056.517971] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16063.992323] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16069.932524] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16077.796104] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16077.797672] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16077.801988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16077.807266] Lustre: Skipped 1 previous similar message [16082.928163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16082.936734] Lustre: Skipped 1 previous similar message [16083.733523] Lustre: server umount lustre-OST0000 complete [16088.035024] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16088.041064] Lustre: Skipped 1 previous similar message [16088.044562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16093.155809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16093.160578] Lustre: Skipped 2 previous similar messages [16094.242887] Lustre: server umount lustre-MDT0000 complete [16098.220810] LustreError: 559166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169106 with bad export cookie 1335773368385653410 [16098.222834] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16098.232134] LustreError: 559166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16098.525582] Lustre: server umount lustre-MDT0001 complete [16111.092348] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [16114.069591] Key type lgssc unregistered [16114.459207] LNet: 562729:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16114.473223] LNetError: 562729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16114.487536] LNet: Removed LNI 192.168.203.107@tcp [16115.379570] Key type .llcrypt unregistered [16115.383818] Key type ._llcrypt unregistered [16139.798618] Key type ._llcrypt registered [16139.810806] Key type .llcrypt registered [16139.931426] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [16140.907735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16141.039188] alg: No test for adler32 (adler32-zlib) [16142.126659] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16142.415619] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [16144.151176] Key type lgssc registered [16145.216033] Lustre: Echo OBD driver; http://www.lustre.org/ [16152.926193] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16152.950745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16154.327865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16158.472769] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16164.838313] LustreError: 564146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16164.862906] LustreError: 564146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16166.105872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16166.377125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16169.427649] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16176.838716] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16183.435500] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16190.693620] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16191.082057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16198.445183] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16200.329229] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [16208.895548] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16216.047839] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16216.054185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16216.086165] Lustre: Skipped 2 previous similar messages [16216.109444] Lustre: Skipped 2 previous similar messages [16217.056528] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.7@tcp (stopping) [16217.812265] Lustre: server umount lustre-MDT0000 complete [16220.289674] LustreError: 564126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.7@tcp arrived at 1788169229 with bad export cookie 716439250524425240 [16220.311986] LustreError: 564126:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16221.158259] LustreError: 564147:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16221.171513] LustreError: 564147:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16222.359573] LustreError: 564128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169231 with bad export cookie 716439250524424344 [16222.368849] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16223.202597] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16223.209079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16223.226777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16226.272711] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16226.274751] LustreError: 564142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16226.333413] LustreError: 564142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16228.660664] Lustre: server umount lustre-OST0000 complete [16232.416367] LustreError: 564146:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16232.442931] LustreError: 564146:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16235.699382] Lustre: server umount lustre-MDT0001 complete [16248.270222] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [16251.311298] Key type lgssc unregistered [16251.623368] LNet: 567834:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16251.633749] LNetError: 567834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16251.654364] LNet: Removed LNI 192.168.203.107@tcp [16252.659846] Key type .llcrypt unregistered [16252.662510] Key type ._llcrypt unregistered [16277.553267] Key type ._llcrypt registered [16277.556613] Key type .llcrypt registered [16277.683285] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [16279.012244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16279.085678] alg: No test for adler32 (adler32-zlib) [16280.037950] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16280.285492] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [16281.992704] Key type lgssc registered [16283.279050] Lustre: Echo OBD driver; http://www.lustre.org/ [16291.990950] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16292.023099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16293.720407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16299.363619] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16304.097983] LustreError: 569254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16304.116172] LustreError: 569254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16309.222710] LustreError: 569253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16310.617589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16311.069166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16315.340529] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16323.387919] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16329.688121] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16337.385576] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16337.782668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16338.815303] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [16345.254550] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16354.287540] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16361.443341] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16361.480708] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [16361.492294] Lustre: Skipped 1 previous similar message [16362.477656] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [16366.101862] Lustre: server umount lustre-MDT0001 complete [16366.570665] LustreError: 569254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16366.583285] LustreError: 569254:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16371.683275] LustreError: 569981:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16371.709595] LustreError: 569981:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16374.753332] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16374.761347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16374.781421] Lustre: Skipped 1 previous similar message [16374.791568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16376.801175] LustreError: 569253:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16376.802566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16379.709923] Lustre: server umount lustre-OST0000 complete [16384.957364] Lustre: server umount lustre-MDT0000 complete [16399.594896] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [16402.372844] Key type lgssc unregistered [16402.701610] LNet: 572937:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16402.709322] LNetError: 572937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16402.735422] LNet: Removed LNI 192.168.203.107@tcp [16403.822768] Key type .llcrypt unregistered [16403.827157] Key type ._llcrypt unregistered [16430.075396] Key type ._llcrypt registered [16430.077297] Key type .llcrypt registered [16430.194587] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [16446.591875] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [16447.830735] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16448.008190] alg: No test for adler32 (adler32-zlib) [16449.141699] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16449.477500] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [16451.151798] Key type lgssc registered [16452.649337] Lustre: Echo OBD driver; http://www.lustre.org/ [16496.136054] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [16513.464540] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16513.522047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16514.730779] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16514.768368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16514.839468] Lustre: lustre-MDT0000: new disk, initializing [16514.994781] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16515.015966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16519.997405] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16535.456039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16535.659457] Lustre: lustre-OST0000: new disk, initializing [16535.663231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16535.666540] Lustre: Skipped 1 previous similar message [16535.670428] Lustre: 577972:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16535.738223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16541.309456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16541.319866] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16541.375635] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16543.052481] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16553.723854] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16571.630283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16571.829874] Lustre: 576954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16571.978906] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16572.126946] Lustre: lustre-MDT0001: new disk, initializing [16572.287541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16572.331984] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [16572.343302] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [16577.798357] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16589.129633] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [16589.348871] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [16593.903881] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [16594.159038] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [16602.074976] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [16602.089428] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16602.110477] Lustre: Skipped 1 previous similar message [16607.168556] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [16612.286426] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.7@tcp (stopping) [16612.301583] Lustre: Skipped 2 previous similar messages [16614.367199] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16614.565153] Lustre: server umount lustre-MDT0001 complete [16622.564806] LustreError: 576962:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16622.580288] LustreError: 576962:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16627.683226] LustreError: 576964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16627.708329] LustreError: 576964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16632.819626] LustreError: 576968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16632.836738] LustreError: 576968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16633.314434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16633.326838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16633.339354] Lustre: Skipped 1 previous similar message [16633.344568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16633.348847] Lustre: Skipped 2 previous similar messages [16637.924570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16637.925685] LustreError: 576967:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16639.244222] Lustre: server umount lustre-OST0000 complete [16643.042090] LustreError: 576968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16658.399132] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16658.703808] Lustre: server umount lustre-MDT0000 complete [16675.326097] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [16678.862748] Key type lgssc unregistered [16679.263447] LNet: 581724:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16679.272411] LNetError: 581724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16679.302599] LNet: Removed LNI 192.168.203.107@tcp [16680.233703] Key type .llcrypt unregistered [16680.237650] Key type ._llcrypt unregistered [16724.965548] Key type ._llcrypt registered [16724.970555] Key type .llcrypt registered [16725.145179] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [16725.446765] Key type .llcrypt unregistered [16725.452800] Key type ._llcrypt unregistered [16743.499574] Key type ._llcrypt registered [16743.505230] Key type .llcrypt registered [16743.664960] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [16744.773250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16745.018027] alg: No test for adler32 (adler32-zlib) [16746.154421] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16746.445516] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [16748.144900] Key type lgssc registered [16749.380120] Lustre: Echo OBD driver; http://www.lustre.org/ [16777.753430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16777.783616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16778.994425] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16779.040841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16779.157409] Lustre: lustre-MDT0000: new disk, initializing [16779.239153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16779.255906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16783.910388] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16795.170664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16795.309554] Lustre: 585103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16795.421386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16795.434073] Lustre: Skipped 1 previous similar message [16795.531229] Lustre: lustre-MDT0001: new disk, initializing [16795.640739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16795.680779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16795.699255] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16799.848154] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16809.414744] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16815.541245] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16822.629818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16822.984274] Lustre: lustre-OST0000: new disk, initializing [16822.988524] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16822.994524] Lustre: 587073:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16823.093286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16829.500165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16829.515474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16829.659671] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16829.740869] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16839.742730] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16850.410495] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16850.416452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16850.425964] Lustre: Skipped 2 previous similar messages [16850.441595] Lustre: Skipped 2 previous similar messages [16853.454852] Lustre: server umount lustre-MDT0000 complete [16855.522470] LustreError: 585109:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16855.567310] LustreError: 585109:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [16857.562383] LustreError: 585898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169866 with bad export cookie 15372088085939591094 [16857.569218] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16857.573937] LustreError: 585898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16858.049656] Lustre: server umount lustre-MDT0001 complete [16871.925715] Lustre: server umount lustre-OST0000 complete [16885.353071] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:51:32 (1788169892) [16893.546910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16894.059173] LustreError: 589101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16894.154343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16898.850464] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16899.558597] LustreError: 589102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16903.656245] LustreError: 589101:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16907.170832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16907.528978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16911.943517] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16920.508736] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16926.409297] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16946.611907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16946.854129] Lustre: Found index 0 for lustre-OST0000, updating log [16947.042973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16953.521518] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [16962.031494] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16964.461717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [16973.801903] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16973.808349] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16973.840166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16974.889902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16974.899733] Lustre: Skipped 2 previous similar messages [16979.252310] Lustre: server umount lustre-OST0000 complete [16996.328948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16996.340871] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16996.352324] Lustre: Skipped 2 previous similar messages [16996.370709] Lustre: Skipped 1 previous similar message [17000.482499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17000.492241] Lustre: Skipped 1 previous similar message [17001.937147] Lustre: server umount lustre-MDT0000 complete [17005.811480] LustreError: 589082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170014 with bad export cookie 15372088085939592487 [17005.812510] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17005.821471] LustreError: 589082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [17006.148415] Lustre: server umount lustre-MDT0001 complete [17009.632502] LNet: 1 local NIs in recovery (showing 1): 192.168.203.107@tcp [17015.311180] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:53:42 (1788170022) [17640.763341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17641.319655] LustreError: 595531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17641.371823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17646.266502] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [17646.563422] LustreError: 595532:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17651.682867] LustreError: 595531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17655.262945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17655.702117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17660.450574] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [17669.658815] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17676.398758] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17696.819399] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17697.093630] Lustre: Found index 0 for lustre-OST0000, updating log [17697.496244] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17700.832755] LNet: 1 local NIs in recovery (showing 1): 192.168.203.107@tcp [17704.575885] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [17713.252521] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17716.667171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17716.682912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17716.685876] Lustre: Skipped 2 previous similar messages [17721.985363] Lustre: server umount lustre-OST0000 complete [17727.456540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17727.476721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17727.501392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17728.546945] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17729.633550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17729.648680] Lustre: Skipped 3 previous similar messages [17733.746231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17733.762551] Lustre: Skipped 7 previous similar messages [17741.279478] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17741.629114] Lustre: server umount lustre-MDT0000 complete [17741.987442] LustreError: 595531:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17742.013840] LustreError: 595531:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17745.696287] LustreError: 595511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170754 with bad export cookie 15372088085939593572 [17745.702177] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17745.707287] LustreError: 595511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17746.001038] Lustre: server umount lustre-MDT0001 complete [17761.247841] LNet: 1 local NIs in recovery (showing 1): 192.168.203.107@tcp [17778.217701] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [17788.622259] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [17829.727295] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:07:17 (1788170837) [17850.098632] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [17853.307274] Key type lgssc unregistered [17853.635761] LNet: 602142:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17853.646651] LNetError: 602142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17958.120930] LNet: Removed LNI 192.168.203.107@tcp [17959.169792] Key type .llcrypt unregistered [17959.172514] Key type ._llcrypt unregistered [17977.163312] Key type ._llcrypt registered [17977.167977] Key type .llcrypt registered [17977.319862] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [17978.808049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17978.833624] alg: No test for adler32 (adler32-zlib) [17979.872365] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17980.103082] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [17981.818672] Key type lgssc registered [17983.047738] Lustre: Echo OBD driver; http://www.lustre.org/ [17988.163065] LNet: Added LNI 192.168.203.107@tcp100 [8/256/0/180] [17989.148764] LNet: Added LNI 192.168.203.107@tcp101 [8/256/0/180] [17990.019672] LNet: Added LNI 192.168.203.107@tcp102 [8/256/0/180] [17991.052625] LNet: Added LNI 192.168.203.107@tcp103 [8/256/0/180] [17992.061046] LNet: Added LNI 192.168.203.107@tcp104 [8/256/0/180] [17993.116360] LNet: Added LNI 192.168.203.107@tcp105 [8/256/0/180] [17994.267302] LNet: Added LNI 192.168.203.107@tcp106 [8/256/0/180] [17995.317751] LNet: Added LNI 192.168.203.107@tcp107 [8/256/0/180] [17996.182562] LNet: Added LNI 192.168.203.107@tcp108 [8/256/0/180] [17997.062287] LNet: Added LNI 192.168.203.107@tcp109 [8/256/0/180] [17997.934421] LNet: Added LNI 192.168.203.107@tcp110 [8/256/0/180] [17998.814050] LNet: Added LNI 192.168.203.107@tcp111 [8/256/0/180] [17999.809979] LNet: Added LNI 192.168.203.107@tcp112 [8/256/0/180] [18000.894047] LNet: Added LNI 192.168.203.107@tcp113 [8/256/0/180] [18001.990761] LNet: Added LNI 192.168.203.107@tcp114 [8/256/0/180] [18002.941430] LNet: Added LNI 192.168.203.107@tcp115 [8/256/0/180] [18003.957037] LNet: Added LNI 192.168.203.107@tcp116 [8/256/0/180] [18004.923567] LNet: Added LNI 192.168.203.107@tcp117 [8/256/0/180] [18005.806981] LNet: Added LNI 192.168.203.107@tcp118 [8/256/0/180] [18006.824290] LNet: Added LNI 192.168.203.107@tcp119 [8/256/0/180] [18007.649837] LNet: Added LNI 192.168.203.107@tcp120 [8/256/0/180] [18008.538991] LNet: Added LNI 192.168.203.107@tcp121 [8/256/0/180] [18009.400915] LNet: Added LNI 192.168.203.107@tcp122 [8/256/0/180] [18010.271927] LNet: Added LNI 192.168.203.107@tcp123 [8/256/0/180] [18011.120328] LNet: Added LNI 192.168.203.107@tcp124 [8/256/0/180] [18011.939380] LNet: Added LNI 192.168.203.107@tcp125 [8/256/0/180] [18012.804990] LNet: Added LNI 192.168.203.107@tcp126 [8/256/0/180] [18013.813857] LNet: Added LNI 192.168.203.107@tcp127 [8/256/0/180] [18014.739905] LNet: Added LNI 192.168.203.107@tcp128 [8/256/0/180] [18015.645816] LNet: Added LNI 192.168.203.107@tcp129 [8/256/0/180] [18016.723963] LNet: Added LNI 192.168.203.107@tcp130 [8/256/0/180] [18017.722718] LNet: Added LNI 192.168.203.107@tcp131 [8/256/0/180] [18018.664372] LNet: Added LNI 192.168.203.107@tcp132 [8/256/0/180] [18019.718750] LNet: Added LNI 192.168.203.107@tcp133 [8/256/0/180] [18020.712690] LNet: Added LNI 192.168.203.107@tcp134 [8/256/0/180] [18021.691873] LNet: Added LNI 192.168.203.107@tcp135 [8/256/0/180] [18029.103826] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18029.142295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18030.277371] Lustre: 605172:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.107@tcp: can't add NID 192.168.203.107@tcp130: rc = -75 [18030.391595] LNetError: 605200:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.107 [18030.421969] LNetError: Protocol error connecting to 192.168.203.107@tcp118 at host 192.168.203.107:988: is it running a compatible version of Lustre? [18030.448745] LNetError: Skipped 6 previous similar messages [18030.538152] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18030.623178] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18030.744800] Lustre: lustre-MDT0000: new disk, initializing [18030.891747] Lustre: 605228:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.107@tcp: can't add NID 192.168.203.107@tcp130: rc = -75 [18030.908530] Lustre: 605228:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18030.936176] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18030.975168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18031.332406] LNet: 7 local NIs in recovery (showing 7): 192.168.203.107@tcp124, 192.168.203.107@tcp125, 192.168.203.107@tcp126, 192.168.203.107@tcp127, 192.168.203.107@tcp128, 192.168.203.107@tcp129, 192.168.203.107@tcp118 [18031.332671] LNetError: 602632:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.107 [18031.366731] LNetError: 602632:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 7 previous similar messages [18031.405250] LNetError: Protocol error connecting to 192.168.203.107@tcp109 at host 192.168.203.107:988: is it running a compatible version of Lustre? [18031.419676] LNetError: Skipped 1 previous similar message [18036.464130] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18048.237243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18048.443917] Lustre: 605187:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18048.534800] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18048.539354] Lustre: Skipped 1 previous similar message [18048.599581] Lustre: lustre-MDT0001: new disk, initializing [18048.640928] Lustre: 605991:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.107@tcp: can't add NID 192.168.203.107@tcp130: rc = -75 [18048.652368] Lustre: 605991:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18048.790625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18048.836721] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18048.853760] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18053.960402] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18057.306590] Lustre: 606365:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.107@tcp: can't add NID 192.168.203.107@tcp130: rc = -75 [18057.319420] Lustre: 606365:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18064.098183] LNetError: 605205:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.107 [18064.106630] LNetError: 605205:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [18064.133203] LNetError: Protocol error connecting to 192.168.203.107@tcp109 at host 192.168.203.107:988: is it running a compatible version of Lustre? [18064.972757] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18068.198296] LNetError: 602631:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.107 [18068.221765] LNetError: Protocol error connecting to 192.168.203.107@tcp123 at host 192.168.203.107:988: is it running a compatible version of Lustre? [18072.026513] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18076.388167] LNetError: 605200:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.107 [18076.401062] LNetError: 605200:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [18076.410233] LNetError: Protocol error connecting to 192.168.203.107@tcp109 at host 192.168.203.107:988: is it running a compatible version of Lustre? [18076.424417] LNetError: Skipped 1 previous similar message [18079.875979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18080.210438] Lustre: lustre-OST0000: new disk, initializing [18080.214391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18080.220214] Lustre: 607166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18080.309083] Lustre: 607171:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.107@tcp: can't add NID 192.168.203.107@tcp130: rc = -75 [18080.319532] Lustre: 607171:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18080.374812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18086.512168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18086.526836] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18086.726484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18087.339776] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18089.701778] LNetError: 605195:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.107 [18089.718452] LNetError: 605195:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18089.742404] LNetError: Protocol error connecting to 192.168.203.107@tcp109 at host 192.168.203.107:988: is it running a compatible version of Lustre? [18089.767419] LNetError: Skipped 2 previous similar messages [18098.904307] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18106.084698] LNetError: 602632:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.107 [18106.091224] LNetError: 602632:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 2 previous similar messages [18106.096633] LNetError: Protocol error connecting to 192.168.203.107@tcp123 at host 192.168.203.107:988: is it running a compatible version of Lustre? [18106.105179] LNetError: Skipped 2 previous similar messages [18107.366312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18107.384129] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18107.389610] LustreError: Skipped 1 previous similar message [18107.395866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18107.399445] Lustre: Skipped 1 previous similar message [18107.437192] Lustre: Skipped 1 previous similar message [18112.489971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18112.497242] Lustre: Skipped 1 previous similar message [18112.985669] Lustre: server umount lustre-OST0000 complete [18117.601683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18117.611137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18117.612547] Lustre: Skipped 1 previous similar message [18117.629689] Lustre: Skipped 1 previous similar message [18122.723672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18122.738563] Lustre: Skipped 1 previous similar message [18123.291790] Lustre: server umount lustre-MDT0000 complete [18127.926258] LustreError: 605181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171136 with bad export cookie 5695656184206187246 [18127.936656] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18127.941698] LustreError: 605181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [18128.295678] Lustre: server umount lustre-MDT0001 complete [18139.940358] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [18143.821585] Key type lgssc unregistered [18144.208556] LNet: 609080:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18144.221680] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18144.232489] LNet: Removed LNI 192.168.203.107@tcp [18145.247937] LNet: Removed LNI 192.168.203.107@tcp100 [18145.249696] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18145.276169] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18145.313107] LNet: Removed LNI 192.168.203.107@tcp101 [18146.337082] LNet: Removed LNI 192.168.203.107@tcp102 [18146.343735] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18146.358014] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18146.372907] LNet: Removed LNI 192.168.203.107@tcp103 [18147.423153] LNet: Removed LNI 192.168.203.107@tcp104 [18148.447118] LNet: Removed LNI 192.168.203.107@tcp105 [18148.453166] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18148.457634] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [18149.471297] LNet: Removed LNI 192.168.203.107@tcp106 [18150.497179] LNet: Removed LNI 192.168.203.107@tcp107 [18151.520024] LNet: Removed LNI 192.168.203.107@tcp108 [18152.543293] LNet: Removed LNI 192.168.203.107@tcp109 [18152.549452] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18152.562815] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [18152.573504] LNet: Removed LNI 192.168.203.107@tcp110 [18153.631242] LNet: Removed LNI 192.168.203.107@tcp111 [18154.656119] LNet: Removed LNI 192.168.203.107@tcp112 [18155.743324] LNet: Removed LNI 192.168.203.107@tcp113 [18156.767493] LNet: Removed LNI 192.168.203.107@tcp114 [18157.791522] LNet: Removed LNI 192.168.203.107@tcp115 [18158.815301] LNet: Removed LNI 192.168.203.107@tcp116 [18159.839197] LNet: Removed LNI 192.168.203.107@tcp117 [18160.863140] LNet: Removed LNI 192.168.203.107@tcp118 [18160.865374] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18160.879244] LNetError: 609080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [18160.894910] LNet: Removed LNI 192.168.203.107@tcp119 [18161.951494] LNet: Removed LNI 192.168.203.107@tcp120 [18162.975255] LNet: Removed LNI 192.168.203.107@tcp121 [18163.999257] LNet: Removed LNI 192.168.203.107@tcp122 [18165.023172] LNet: Removed LNI 192.168.203.107@tcp123 [18166.047154] LNet: Removed LNI 192.168.203.107@tcp124 [18167.071318] LNet: Removed LNI 192.168.203.107@tcp125 [18168.095146] LNet: Removed LNI 192.168.203.107@tcp126 [18169.119166] LNet: Removed LNI 192.168.203.107@tcp127 [18170.143191] LNet: Removed LNI 192.168.203.107@tcp128 [18171.167848] LNet: Removed LNI 192.168.203.107@tcp129 [18171.171205] LNet: Removed LNI 192.168.203.107@tcp130 [18171.177990] LNet: Removed LNI 192.168.203.107@tcp131 [18171.182359] LNet: Removed LNI 192.168.203.107@tcp132 [18171.190905] LNet: Removed LNI 192.168.203.107@tcp133 [18171.196595] LNet: Removed LNI 192.168.203.107@tcp134 [18171.205760] LNet: Removed LNI 192.168.203.107@tcp135 [18172.116754] Key type .llcrypt unregistered [18172.125617] Key type ._llcrypt unregistered [18197.215744] Key type ._llcrypt registered [18197.219301] Key type .llcrypt registered [18197.337951] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [18211.808762] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [18212.915254] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18212.954240] alg: No test for adler32 (adler32-zlib) [18214.000960] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18214.238066] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [18215.911162] Key type lgssc registered [18216.926972] Lustre: Echo OBD driver; http://www.lustre.org/ [18252.189418] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:14:20 (1788171260) [18272.678736] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [18275.813814] Key type lgssc unregistered [18276.226703] LNet: 613168:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18276.233098] LNetError: 613168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18276.269798] LNet: Removed LNI 192.168.203.107@tcp [18277.342511] Key type .llcrypt unregistered [18277.346077] Key type ._llcrypt unregistered [18298.052868] Key type ._llcrypt registered [18298.061308] Key type .llcrypt registered [18298.209411] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [18299.536117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18299.601103] alg: No test for adler32 (adler32-zlib) [18300.707786] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18300.896818] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [18302.567467] Key type lgssc registered [18303.867259] Lustre: Echo OBD driver; http://www.lustre.org/ [18314.484507] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18314.552836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18315.906198] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18315.970849] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18316.090989] Lustre: lustre-MDT0000: new disk, initializing [18316.181577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18316.194876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18321.478162] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18333.395978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18333.519859] Lustre: 614383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18333.584503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18333.598670] Lustre: Skipped 1 previous similar message [18333.711727] Lustre: lustre-MDT0001: new disk, initializing [18333.857060] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18333.898429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18333.931442] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18338.488909] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18348.919244] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18355.266852] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18363.854561] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18364.223599] Lustre: lustre-OST0000: new disk, initializing [18364.227085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18364.233662] Lustre: 616351:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18364.300125] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18370.607772] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18370.620167] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18370.785633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18370.891431] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18382.649271] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18387.725754] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing lnet_if_list [18393.809676] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [18393.838280] LNet: Added LNI 192.168.203.107@tcp10 [8/256/0/180] [18400.479567] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [18400.537176] LNet: Added LNI 192.168.203.107@tcp11 [8/256/0/180] [18406.062762] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [18406.100474] LNet: Added LNI 192.168.203.107@tcp12 [8/256/0/180] [18412.264642] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [18412.361210] LNet: Added LNI 192.168.203.107@tcp13 [8/256/0/180] [18419.007479] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [18419.062473] LNet: Added LNI 192.168.203.107@tcp14 [8/256/0/180] [18425.658582] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [18425.705580] LNet: Added LNI 192.168.203.107@tcp15 [8/256/0/180] [18431.705213] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [18431.732277] LNet: Added LNI 192.168.203.107@tcp16 [8/256/0/180] [18437.231135] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [18437.273101] LNet: Added LNI 192.168.203.107@tcp17 [8/256/0/180] [18442.569921] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [18442.623627] LNet: Added LNI 192.168.203.107@tcp18 [8/256/0/180] [18449.949464] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [18450.007439] LNet: Added LNI 192.168.203.107@tcp19 [8/256/0/180] [18456.013550] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [18456.063052] LNet: Added LNI 192.168.203.107@tcp20 [8/256/0/180] [18461.728748] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [18461.763883] LNet: Added LNI 192.168.203.107@tcp21 [8/256/0/180] [18468.231136] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [18468.261086] LNet: Added LNI 192.168.203.107@tcp22 [8/256/0/180] [18473.740594] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [18473.762657] LNet: Added LNI 192.168.203.107@tcp23 [8/256/0/180] [18479.906139] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [18479.947785] LNet: Added LNI 192.168.203.107@tcp24 [8/256/0/180] [18487.106893] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [18487.162842] LNet: Added LNI 192.168.203.107@tcp25 [8/256/0/180] [18493.861568] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [18493.885651] LNet: Added LNI 192.168.203.107@tcp26 [8/256/0/180] [18499.906794] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [18499.975671] LNet: Added LNI 192.168.203.107@tcp27 [8/256/0/180] [18506.425130] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [18506.476191] LNet: Added LNI 192.168.203.107@tcp28 [8/256/0/180] [18513.383810] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [18513.428766] LNet: Added LNI 192.168.203.107@tcp29 [8/256/0/180] [18520.613873] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [18520.644488] LNet: Added LNI 192.168.203.107@tcp30 [8/256/0/180] [18527.312968] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [18527.356407] LNet: Added LNI 192.168.203.107@tcp31 [8/256/0/180] [18533.558932] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [18533.588922] LNet: Added LNI 192.168.203.107@tcp32 [8/256/0/180] [18540.106540] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [18540.153336] LNet: Added LNI 192.168.203.107@tcp33 [8/256/0/180] [18547.568823] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [18547.619850] LNet: Added LNI 192.168.203.107@tcp34 [8/256/0/180] [18554.886116] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [18554.936730] LNet: Added LNI 192.168.203.107@tcp35 [8/256/0/180] [18562.310752] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [18562.337452] LNet: Added LNI 192.168.203.107@tcp36 [8/256/0/180] [18571.481731] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [18571.518086] LNet: Added LNI 192.168.203.107@tcp37 [8/256/0/180] [18580.857939] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [18580.891386] LNet: Added LNI 192.168.203.107@tcp38 [8/256/0/180] [18589.861027] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [18589.896213] LNet: Added LNI 192.168.203.107@tcp39 [8/256/0/180] [18598.608256] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [18598.633817] LNet: Added LNI 192.168.203.107@tcp40 [8/256/0/180] [18607.304299] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [18607.357950] LNet: Added LNI 192.168.203.107@tcp41 [8/256/0/180] [18615.639389] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [18615.684409] LNet: Added LNI 192.168.203.107@tcp42 [8/256/0/180] [18624.932256] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [18624.966703] LNet: Added LNI 192.168.203.107@tcp43 [8/256/0/180] [18632.850709] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [18632.903950] LNet: Added LNI 192.168.203.107@tcp44 [8/256/0/180] [18642.219181] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [18642.269625] LNet: Added LNI 192.168.203.107@tcp45 [8/256/0/180] [18651.139344] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [18651.177145] LNet: Added LNI 192.168.203.107@tcp46 [8/256/0/180] [18661.323767] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [18661.351267] LNet: Added LNI 192.168.203.107@tcp47 [8/256/0/180] [18671.336504] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [18671.374421] LNet: Added LNI 192.168.203.107@tcp48 [8/256/0/180] [18681.094886] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [18681.125212] LNet: Added LNI 192.168.203.107@tcp49 [8/256/0/180] [18690.606594] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [18690.632150] LNet: Added LNI 192.168.203.107@tcp50 [8/256/0/180] [18708.961944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18708.969165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18708.974389] Lustre: Skipped 2 previous similar messages [18708.989818] Lustre: Skipped 2 previous similar messages [18714.083365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18714.088646] Lustre: Skipped 2 previous similar messages [18716.301833] Lustre: server umount lustre-MDT0000 complete [18719.982079] LustreError: 615186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171728 with bad export cookie 5872897570392359062 [18719.982965] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18719.993016] LustreError: 615186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [18720.365740] Lustre: server umount lustre-MDT0001 complete [18739.003417] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [18740.703720] Lustre: 613651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171733/real 1788171733] req@0000000030d7eb94 x1875033518688640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171749 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18740.746519] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18745.887983] Lustre: 613651:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171738/real 1788171738] req@000000002d96bdfa x1875033518688896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171754 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18748.624485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18750.943517] Lustre: 613653:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171743/real 1788171743] req@00000000d7666c65 x1875033518689024/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18756.063158] Lustre: 613654:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788171748/real 1788171748] req@00000000882c8df6 x1875033518689152/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788171764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18764.320151] LustreError: 613650:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ba7412cb x1875033518689792/t0(0) o250->MGC192.168.203.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [18764.601663] LustreError: 616345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18764.618352] LustreError: 616345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [18764.657636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18765.666764] LustreError: 616346:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18765.686758] LustreError: 616346:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18769.120335] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18770.918494] LustreError: 640599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18770.926189] LustreError: 640599:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18775.015341] LustreError: 616345:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18775.034646] LustreError: 616345:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18775.037874] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [18778.482349] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18778.787850] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18783.543976] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18784.242590] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [18791.190543] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18796.339980] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18814.956468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18814.957656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18814.974087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18820.064481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18820.072344] Lustre: Skipped 2 previous similar messages [18820.669631] Lustre: server umount lustre-OST0000 complete [18825.189347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18825.196541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18825.208682] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18825.216069] Lustre: Skipped 2 previous similar messages [18825.257368] Lustre: Skipped 1 previous similar message [18831.162117] Lustre: server umount lustre-MDT0000 complete [18835.021286] LustreError: 614375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171843 with bad export cookie 5872897570392361708 [18835.025569] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18835.031624] LustreError: 614375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [18835.314604] Lustre: server umount lustre-MDT0001 complete [18846.582552] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [18849.799580] Key type lgssc unregistered [18850.277912] LNet: 643378:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18850.293634] LNetError: 643378:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18850.301104] LNet: Removed LNI 192.168.203.107@tcp [18850.303537] LNet: Removed LNI 192.168.203.107@tcp10 [18850.305559] LNet: Removed LNI 192.168.203.107@tcp11 [18850.311522] LNet: Removed LNI 192.168.203.107@tcp12 [18850.314752] LNet: Removed LNI 192.168.203.107@tcp13 [18850.316755] LNet: Removed LNI 192.168.203.107@tcp14 [18850.321370] LNet: Removed LNI 192.168.203.107@tcp15 [18850.325429] LNet: Removed LNI 192.168.203.107@tcp16 [18850.328376] LNet: Removed LNI 192.168.203.107@tcp17 [18850.331306] LNet: Removed LNI 192.168.203.107@tcp18 [18850.335650] LNet: Removed LNI 192.168.203.107@tcp19 [18850.338599] LNet: Removed LNI 192.168.203.107@tcp20 [18850.342787] LNet: Removed LNI 192.168.203.107@tcp21 [18850.347862] LNet: Removed LNI 192.168.203.107@tcp22 [18850.351393] LNet: Removed LNI 192.168.203.107@tcp23 [18850.356226] LNet: Removed LNI 192.168.203.107@tcp24 [18850.359064] LNet: Removed LNI 192.168.203.107@tcp25 [18850.364264] LNet: Removed LNI 192.168.203.107@tcp26 [18850.367256] LNet: Removed LNI 192.168.203.107@tcp27 [18850.370341] LNet: Removed LNI 192.168.203.107@tcp28 [18850.374994] LNet: Removed LNI 192.168.203.107@tcp29 [18850.379439] LNet: Removed LNI 192.168.203.107@tcp30 [18850.381778] LNet: Removed LNI 192.168.203.107@tcp31 [18850.389131] LNet: Removed LNI 192.168.203.107@tcp32 [18850.391943] LNet: Removed LNI 192.168.203.107@tcp33 [18850.397846] LNet: Removed LNI 192.168.203.107@tcp34 [18850.399933] LNet: Removed LNI 192.168.203.107@tcp35 [18850.406056] LNet: Removed LNI 192.168.203.107@tcp36 [18850.407990] LNet: Removed LNI 192.168.203.107@tcp37 [18850.413989] LNet: Removed LNI 192.168.203.107@tcp38 [18850.416081] LNet: Removed LNI 192.168.203.107@tcp39 [18850.422384] LNet: Removed LNI 192.168.203.107@tcp40 [18850.426128] LNet: Removed LNI 192.168.203.107@tcp41 [18850.429683] LNet: Removed LNI 192.168.203.107@tcp42 [18850.433291] LNet: Removed LNI 192.168.203.107@tcp43 [18850.436355] LNet: Removed LNI 192.168.203.107@tcp44 [18850.441557] LNet: Removed LNI 192.168.203.107@tcp45 [18850.447680] LNet: Removed LNI 192.168.203.107@tcp46 [18850.451323] LNet: Removed LNI 192.168.203.107@tcp47 [18850.454347] LNet: Removed LNI 192.168.203.107@tcp48 [18850.456829] LNet: Removed LNI 192.168.203.107@tcp49 [18850.466542] LNet: Removed LNI 192.168.203.107@tcp50 [18851.382223] Key type .llcrypt unregistered [18851.384201] Key type ._llcrypt unregistered [18874.187519] Key type ._llcrypt registered [18874.192206] Key type .llcrypt registered [18874.308331] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [18888.644440] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [18889.970946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18889.997559] alg: No test for adler32 (adler32-zlib) [18891.100213] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18891.413183] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [18893.103201] Key type lgssc registered [18894.323444] Lustre: Echo OBD driver; http://www.lustre.org/ [18929.699660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18929.729204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18931.169370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18931.227422] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18931.395112] Lustre: lustre-MDT0000: new disk, initializing [18931.535630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18931.565759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18936.128597] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18948.446626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18948.605433] Lustre: 646581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18948.653822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18948.658417] Lustre: Skipped 1 previous similar message [18948.735880] Lustre: lustre-MDT0001: new disk, initializing [18948.840154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18948.888208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18948.914699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18954.140519] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18964.104355] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18969.987788] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18976.900273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18977.068202] Lustre: *** cfs_fail_loc=90f, val=0*** [18977.156304] Lustre: lustre-OST0000: new disk, initializing [18977.163357] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18977.172427] Lustre: 648598:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18977.278385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18983.294037] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [18985.503210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18985.514594] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18985.621824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18991.371242] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19000.808716] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19000.827746] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19000.845812] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19001.318581] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19004.200946] Lustre: server umount lustre-OST0000 complete [19010.529150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19010.535256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19010.550979] Lustre: Skipped 1 previous similar message [19010.561700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19010.568029] Lustre: Skipped 2 previous similar messages [19011.553092] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19011.581881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19011.590828] Lustre: Skipped 1 previous similar message [19013.847850] Lustre: server umount lustre-MDT0000 complete [19017.325235] LustreError: 647384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172026 with bad export cookie 2429886483009565574 [19017.332816] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19017.344640] LustreError: 647384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19017.667672] Lustre: server umount lustre-MDT0001 complete [19028.421398] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [19031.372825] Key type lgssc unregistered [19031.649624] LNet: 650503:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19031.655617] LNetError: 650503:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19031.669417] LNet: Removed LNI 192.168.203.107@tcp [19032.449725] Key type .llcrypt unregistered [19032.451584] Key type ._llcrypt unregistered [19051.840658] Key type ._llcrypt registered [19051.843262] Key type .llcrypt registered [19051.933261] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [19053.192169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19053.215235] alg: No test for adler32 (adler32-zlib) [19054.268139] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19054.449202] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [19056.078899] Key type lgssc registered [19056.979908] Lustre: Echo OBD driver; http://www.lustre.org/ [19063.949493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19063.979067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19065.532584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19069.906397] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [19076.069184] LustreError: 651774:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19076.091623] LustreError: 651774:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [19078.180351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19078.612605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19082.761866] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [19090.340750] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19096.264215] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19103.507582] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19103.939792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19110.852876] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [19119.275176] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19128.383162] loop0: detected capacity change from 0 to 204800000 [19128.537053] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19128.556052] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19128.568312] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19134.733279] loop0: detected capacity change from 0 to 204800000 [19134.949906] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19134.963140] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19134.980714] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [19140.248782] loop0: detected capacity change from 0 to 204800000 [19146.267932] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19146.400562] Lustre: fs15246-OST0000: new disk, initializing [19146.420262] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [19146.431992] Lustre: 655368:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19146.515841] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19153.269140] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [19163.189191] loop1: detected capacity change from 0 to 204800000 [19169.835602] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19169.987307] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [19170.042529] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [19170.123776] Lustre: fs15246-MDT0000: new disk, initializing [19170.352621] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19170.380753] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19170.511783] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [19171.553281] Lustre: 656489:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000004a92e415 x1875034309180544/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788172196 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [19175.790934] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [19176.678176] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [19176.695061] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [19176.748058] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [19201.511136] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19201.529322] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [19204.061568] Lustre: server umount fs15246-MDT0000 complete [19206.637533] LustreError: 651769:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19211.782891] LustreError: 651770:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19215.503318] Lustre: server umount fs15246-OST0000 complete [19229.163573] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19229.171258] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19229.191570] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19232.233154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19232.233198] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19235.300031] Lustre: server umount lustre-OST0000 complete [19242.464812] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19242.469048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19242.471986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19242.487394] Lustre: Skipped 1 previous similar message [19242.517706] Lustre: Skipped 2 previous similar messages [19245.878362] Lustre: server umount lustre-MDT0000 complete [19247.591717] LustreError: 657978:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19247.620505] LustreError: 657978:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19249.768045] LustreError: 658837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172258 with bad export cookie 12323267690519064854 [19249.779882] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19249.782864] LustreError: 658837:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [19250.147993] Lustre: server umount lustre-MDT0001 complete [19259.994782] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [19262.796933] Key type lgssc unregistered [19263.106846] LNet: 659588:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19263.113448] LNetError: 659588:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19263.134695] LNet: Removed LNI 192.168.203.107@tcp [19264.213343] Key type .llcrypt unregistered [19264.221328] Key type ._llcrypt unregistered [19283.644490] Key type ._llcrypt registered [19283.649132] Key type .llcrypt registered [19283.749371] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [19284.938085] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19284.994289] alg: No test for adler32 (adler32-zlib) [19285.998753] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19286.358949] LNet: Added LNI 192.168.203.107@tcp [8/256/0/180] [19288.039433] Key type lgssc registered [19289.123474] Lustre: Echo OBD driver; http://www.lustre.org/ [19327.495996] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [19337.709702] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [19376.349063] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19376.389042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19377.716339] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19377.756987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19377.879342] Lustre: lustre-MDT0000: new disk, initializing [19378.002607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19378.061835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19382.335799] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [19392.752619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19392.950534] Lustre: 664057:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19393.069260] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19393.072889] Lustre: Skipped 1 previous similar message [19393.150838] Lustre: lustre-MDT0001: new disk, initializing [19393.309145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19393.357835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19393.373024] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19399.248843] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [19409.802483] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19416.109245] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19424.136278] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19424.487661] Lustre: lustre-OST0000: new disk, initializing [19424.493992] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19424.504682] Lustre: 666025:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19424.596828] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19429.963523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19429.973516] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19430.121093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19430.700454] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_default_debug -1 all [19440.309460] Lustre: DEBUG MARKER: oleg307-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19445.381563] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [19445.599633] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [19450.143703] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [19450.403428] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [19455.968435] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19455.972539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19455.976521] LustreError: Skipped 1 previous similar message [19455.983528] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19455.992459] Lustre: Skipped 1 previous similar message [19458.827234] Lustre: server umount lustre-OST0000 complete [19465.186092] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19465.202208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19465.226473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19465.235550] Lustre: Skipped 1 previous similar message [19466.209735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19466.230474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19466.233457] Lustre: Skipped 1 previous similar message [19469.577897] Lustre: server umount lustre-MDT0000 complete [19473.580592] LustreError: 664852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172482 with bad export cookie 9300722006201486350 [19473.583995] LustreError: MGC192.168.203.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19473.593906] LustreError: 664852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [19474.225111] Lustre: server umount lustre-MDT0001 complete [19498.838852] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing set_hostid [19507.037734] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing load_modules_local [19519.375917] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 19260 sec ========= 06:35:28 (1788172528) [19519.967927] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:35:28 (1788172528) === [19521.306614] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:35:29 (1788172529) === [19535.665082] Lustre: DEBUG MARKER: oleg307-server.virtnet: executing unload_modules_local [19536.668495] Key type lgssc unregistered [19536.797241] LNet: 672661:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19536.801070] LNetError: 672661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19536.807290] LNet: Removed LNI 192.168.203.107@tcp [19537.094569] Key type .llcrypt unregistered [19537.095440] Key type ._llcrypt unregistered