[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 500800474 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002318] x2apic enabled [ 0.003005] Switched APIC routing to physical x2apic. [ 0.004008] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007019] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008006] pid_max: default: 32768 minimum: 301 [ 0.010047] LSM: Security Framework initializing [ 0.011031] Yama: becoming mindful. [ 0.012022] SELinux: Initializing. [ 0.013058] *** VALIDATE selinux *** [ 0.020703] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024157] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027067] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028081] *** VALIDATE tmpfs *** [ 0.029422] *** VALIDATE proc *** [ 0.031109] *** VALIDATE cgroup *** [ 0.032005] *** VALIDATE cgroup2 *** [ 0.033216] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034123] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036022] Spectre V2 : User space: Vulnerable [ 0.037005] Speculative Store Bypass: Vulnerable [ 0.040286] debug: unmapping init [mem 0xffffffffa9259000-0xffffffffa9260fff] [ 0.042151] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043583] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044016] ... version: 2 [ 0.045008] ... bit width: 48 [ 0.046006] ... generic registers: 4 [ 0.047007] ... value mask: 0000ffffffffffff [ 0.048006] ... max period: 00007fffffffffff [ 0.049013] ... fixed-purpose events: 3 [ 0.050007] ... event mask: 000000070000000f [ 0.052220] rcu: Hierarchical SRCU implementation. [ 0.054329] smp: Bringing up secondary CPUs ... [ 0.055477] x86: Booting SMP configuration: [ 0.056013] .... node #0, CPUs: #1 #2 #3 [ 0.063023] smp: Brought up 1 node, 4 CPUs [ 0.065016] smpboot: Max logical packages: 1 [ 0.066008] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.250027] node 0 deferred pages initialised in 181ms [ 0.253234] devtmpfs: initialized [ 0.254222] x86/mm: Memory block size: 128MB [ 0.260286] gcov: version magic: 0x41383552 [ 0.266312] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.268089] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.269462] pinctrl core: initialized pinctrl subsystem [ 0.271193] [ 0.271656] ************************************************************* [ 0.273011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.275008] ** ** [ 0.276010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.277009] ** ** [ 0.279009] ** This means that this kernel is built to expose internal ** [ 0.280008] ** IOMMU data structures, which may compromise security on ** [ 0.282012] ** your system. ** [ 0.284011] ** ** [ 0.285010] ** If you see this message and you are not debugging the ** [ 0.287011] ** kernel, report this immediately to your vendor! ** [ 0.289013] ** ** [ 0.291012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.293013] ************************************************************* [ 0.294000] NET: Registered protocol family 16 [ 0.295528] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.298087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.300062] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.303137] cpuidle: using governor menu [ 0.304748] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.307392] PCI: Using configuration type 1 for base access [ 0.309120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.317012] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.319025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.322004] cryptd: max_cpu_qlen set to 1000 [ 0.324283] ACPI: Added _OSI(Module Device) [ 0.325029] ACPI: Added _OSI(Processor Device) [ 0.326024] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.327014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.332590] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.338480] ACPI: Interpreter enabled [ 0.340060] ACPI: PM: (supports S0 S3 S4 S5) [ 0.342017] ACPI: Using IOAPIC for interrupt routing [ 0.344172] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.347472] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.356109] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.358020] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.360011] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.362060] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.366987] acpiphp: Slot [2] registered [ 0.369168] acpiphp: Slot [5] registered [ 0.370091] acpiphp: Slot [6] registered [ 0.371094] acpiphp: Slot [7] registered [ 0.373098] acpiphp: Slot [8] registered [ 0.374039] acpiphp: Slot [9] registered [ 0.374919] acpiphp: Slot [10] registered [ 0.376070] acpiphp: Slot [3] registered [ 0.376913] acpiphp: Slot [4] registered [ 0.377042] acpiphp: Slot [11] registered [ 0.377810] acpiphp: Slot [12] registered [ 0.379048] acpiphp: Slot [13] registered [ 0.380045] acpiphp: Slot [14] registered [ 0.380993] acpiphp: Slot [15] registered [ 0.382078] acpiphp: Slot [16] registered [ 0.383094] acpiphp: Slot [17] registered [ 0.383985] acpiphp: Slot [18] registered [ 0.385052] acpiphp: Slot [19] registered [ 0.385946] acpiphp: Slot [20] registered [ 0.387070] acpiphp: Slot [21] registered [ 0.388046] acpiphp: Slot [22] registered [ 0.390078] acpiphp: Slot [23] registered [ 0.391075] acpiphp: Slot [24] registered [ 0.393215] acpiphp: Slot [25] registered [ 0.394053] acpiphp: Slot [26] registered [ 0.395095] acpiphp: Slot [27] registered [ 0.397098] acpiphp: Slot [28] registered [ 0.398091] acpiphp: Slot [29] registered [ 0.400073] acpiphp: Slot [30] registered [ 0.401085] acpiphp: Slot [31] registered [ 0.402054] PCI host bridge to bus 0000:00 [ 0.403015] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.404021] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.407025] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.408021] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.411028] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.414042] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.417198] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.421030] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.424283] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.430560] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.436117] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.438015] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.440010] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.443016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.445384] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.447730] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.452037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.453727] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.458010] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.470917] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.476000] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.480596] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.487017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.492010] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.511016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.524205] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.533017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.541016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.557020] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.564641] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.571010] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.580015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.594012] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.606521] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.612016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.618015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.629015] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.635000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.644017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.648014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.658023] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.670751] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.677014] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.688024] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.708013] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.719456] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.721439] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.724338] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.727026] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.730211] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.736212] iommu: Default domain type: Passthrough [ 0.737510] SCSI subsystem initialized [ 0.739145] ACPI: bus type USB registered [ 0.740214] usbcore: registered new interface driver usbfs [ 0.742090] usbcore: registered new interface driver hub [ 0.744074] usbcore: registered new device driver usb [ 0.746156] pps_core: LinuxPPS API ver. 1 registered [ 0.747009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.750056] PTP clock support registered [ 0.752151] EDAC MC: Ver: 3.0.0 [ 0.753115] PCI: Using ACPI for IRQ routing [ 0.754702] NetLabel: Initializing [ 0.755006] NetLabel: domain hash size = 128 [ 0.755802] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.757051] NetLabel: unlabeled traffic allowed by default [ 0.759206] vgaarb: loaded [ 0.760268] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.761007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.766133] clocksource: Switched to clocksource kvm-clock [ 0.874703] VFS: Disk quotas dquot_6.6.0 [ 0.876356] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.878813] *** VALIDATE ramfs *** [ 0.880157] *** VALIDATE hugetlbfs *** [ 0.881691] pnp: PnP ACPI init [ 0.883643] pnp: PnP ACPI: found 6 devices [ 0.899128] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.901620] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.903515] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.905241] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.907277] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.909251] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.911591] NET: Registered protocol family 2 [ 0.913625] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.918153] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.921359] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.924954] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.927312] TCP: Hash tables configured (established 65536 bind 65536) [ 0.929128] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.930933] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.932650] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.934494] NET: Registered protocol family 1 [ 0.936149] RPC: Registered named UNIX socket transport module. [ 0.937411] RPC: Registered udp transport module. [ 0.938508] RPC: Registered tcp transport module. [ 0.939547] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.940848] NET: Registered protocol family 44 [ 0.942606] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.944478] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.946645] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.949206] PCI: CLS 0 bytes, default 64 [ 0.950921] Unpacking initramfs... [ 2.584402] debug: unmapping init [mem 0xffffa0d43cc54000-0xffffa0d43ffbffff] [ 2.588663] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.590822] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.593573] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.812831] Initialise system trusted keyrings [ 3.814361] Key type blacklist registered [ 3.817261] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.826571] zbud: loaded [ 3.830814] *** VALIDATE nfs *** [ 3.832962] *** VALIDATE nfs4 *** [ 3.836489] pstore: using deflate compression [ 3.845635] Platform Keyring initialized [ 3.982657] NET: Registered protocol family 38 [ 3.984301] Key type asymmetric registered [ 3.985658] Asymmetric key parser 'x509' registered [ 3.987503] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.990373] io scheduler mq-deadline registered [ 3.992123] io scheduler kyber registered [ 3.993988] io scheduler bfq registered [ 3.995781] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.000315] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.002702] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.005341] ACPI: Power Button [PWRF] [ 4.011764] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.019370] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.034487] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.042197] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.058478] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.084307] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.113372] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.118406] Non-volatile memory driver v1.3 [ 4.119884] Linux agpgart interface v0.103 [ 4.150835] virtio_blk virtio1: [vda] 145160 512-byte logical blocks (74.3 MB/70.9 MiB) [ 4.153634] vda: detected capacity change from 0 to 74321920 [ 4.196189] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.199317] vdb: detected capacity change from 0 to 1073741824 [ 4.213809] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.217279] vdc: detected capacity change from 0 to 2621440000 [ 4.249569] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.253797] vdd: detected capacity change from 0 to 2621440000 [ 4.282319] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.285313] vde: detected capacity change from 0 to 4294967296 [ 4.309755] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.313333] vdf: detected capacity change from 0 to 4294967296 [ 4.322908] libphy: Fixed MDIO Bus: probed [ 4.339386] usbcore: registered new interface driver usbserial_generic [ 4.344164] usbserial: USB Serial support registered for generic [ 4.346714] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.350856] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.352793] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.356111] mousedev: PS/2 mouse device common for all mice [ 4.359281] rtc_cmos 00:05: RTC can wake from S4 [ 4.360282] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.367440] rtc_cmos 00:05: registered as rtc0 [ 4.369798] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.373484] intel_pstate: CPU model not supported [ 4.377650] hid: raw HID events driver (C) Jiri Kosina [ 4.378840] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 4.381141] usbcore: registered new interface driver usbhid [ 4.381146] usbhid: USB HID core driver [ 4.381359] drop_monitor: Initializing network drop monitor service [ 4.381484] Initializing XFRM netlink socket [ 4.381867] NET: Registered protocol family 10 [ 4.383431] Segment Routing with IPv6 [ 4.399923] NET: Registered protocol family 17 [ 4.401411] mpls_gso: MPLS GSO support [ 4.405715] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 4.409675] RAS: Correctable Errors collector initialized. [ 4.412906] AVX version of gcm_enc/dec engaged. [ 4.414159] AES CTR mode by8 optimization enabled [ 4.498469] sched_clock: Marking stable (4498392563, 0)->(5344793197, -846400634) [ 4.502443] registered taskstats version 1 [ 4.504521] Loading compiled-in X.509 certificates [ 4.506727] zswap: loaded using pool lzo/zbud [ 4.534293] Key type big_key registered [ 4.546474] Key type encrypted registered [ 4.547806] ima: No TPM chip found, activating TPM-bypass! [ 4.549589] ima: Allocated hash algorithm: sha1 [ 4.551456] ima: No architecture policies found [ 4.553286] evm: Initialising EVM extended attributes: [ 4.555214] evm: security.selinux [ 4.556212] evm: security.ima [ 4.556795] evm: security.capability [ 4.558111] evm: HMAC attrs: 0x1 [ 4.560342] rtc_cmos 00:05: setting system clock to 2026-06-25 04:15:37 UTC (1782360937) [ 4.571780] debug: unmapping init [mem 0xffffffffaa203000-0xffffffffaa3fffff] [ 4.574577] debug: unmapping init [mem 0xffffffffa8f82000-0xffffffffa9258fff] [ 4.586256] Write protecting the kernel read-only data: 28672k [ 4.589159] debug: unmapping init [mem 0xffffffffa7603000-0xffffffffa77fffff] [ 4.591413] debug: unmapping init [mem 0xffffffffa7f14000-0xffffffffa7ffffff] [ 4.625560] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.633537] systemd[1]: Detected virtualization kvm. [ 4.635093] systemd[1]: Detected architecture x86-64. [ 4.636738] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.663879] systemd[1]: No hostname configured. [ 4.666468] systemd[1]: Set hostname to . [ 4.668475] random: systemd: uninitialized urandom read (16 bytes read) [ 4.670870] systemd[1]: Initializing machine ID from random generator. [ 4.989376] random: ln: uninitialized urandom read (6 bytes read) [ 5.277679] random: systemd: uninitialized urandom read (16 bytes read) [ 5.280995] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 5.287262] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.293829] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Started Memstrack Anylazing Service. [ 5.498439] hrtimer: interrupt took 6391882 ns Starting Setup Virtual Console... [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.344554] device-mapper: uevent: version 1.0.3 [ 6.347097] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 7.495021] virtio_net virtio0 ens2: renamed from eth0 [ 7.594349] random: fast init done [ 7.678097] scsi host0: ata_piix [ 7.781403] scsi host1: ata_piix [ 7.795525] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 7.815430] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.066664] random: crng init done [ 13.075343] random: 7 urandom warning(s) missed due to ratelimiting [ 14.237592] dracut-initqueue[585]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 15.675917] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 17.256226] printk: systemd: 25 output lines suppressed due to ratelimiting [ 17.691504] SELinux: Disabled at runtime. [ 17.863158] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 17.872575] systemd[1]: Detected virtualization kvm. [ 17.874580] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 18.751467] systemd[1]: initrd-switch-root.service: Succeeded. [ 18.755349] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 18.763551] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 18.769919] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 18.776561] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 18.785389] systemd[1]: Starting Journal Service... Starting Journal Service... [ 18.796912] systemd[1]: Listening on RPCbind Server Activation Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Created slice system-getty.slice. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. Starting Create list of required st…ce nodes for the current kernel... Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ 18.892287] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. Mounting Huge Pages File System... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 19.959587] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 20.638110] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 20.638946] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 20.815847] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 20.867981] EDAC sbridge: Ver: 1.1.2 [ 24.161066] Key type dns_resolver registered [ 24.613827] NFS: Registering the id_resolver key type [ 24.615942] Key type id_resolver registered [ 24.617591] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg141-server login: [ 55.830539] libcfs: loading out-of-tree module taints kernel. [ 55.849527] Key type ._llcrypt registered [ 55.851135] Key type .llcrypt registered [ 55.891743] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 62.637226] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 63.159796] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.165575] alg: No test for adler32 (adler32-zlib) [ 64.117563] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 64.425560] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 66.031144] Key type lgssc registered [ 66.564367] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.388866] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.761835] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 91.276439] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 91.285965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 92.365384] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 92.376817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 92.412724] Lustre: lustre-MDT0000: new disk, initializing [ 92.432343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 92.439579] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 93.706431] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 98.787922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 98.823583] Lustre: 6478:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 98.834485] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 98.836493] Lustre: Skipped 1 previous similar message [ 98.868690] Lustre: lustre-MDT0001: new disk, initializing [ 98.887772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 98.895617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 98.898630] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 100.176773] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 102.504345] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 105.588088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 105.669077] Lustre: lustre-OST0000: new disk, initializing [ 105.671228] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 105.673150] Lustre: 8410:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 105.689064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 107.544239] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 107.628178] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 107.630332] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 107.651880] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 112.687969] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 112.724714] Lustre: lustre-OST0001: new disk, initializing [ 112.726604] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 112.728845] Lustre: 9488:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 112.747148] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 114.492236] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 115.242790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 115.245560] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 115.259908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 120.330325] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 126.299651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 128.543501] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing check_logdir /tmp/testlogs/ [ 130.106327] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing yml_node [ 131.666651] Lustre: DEBUG MARKER: Client: 2.17.54.83 [ 132.581889] Lustre: DEBUG MARKER: MDS: 2.17.54.83 [ 133.434122] Lustre: DEBUG MARKER: OSS: 2.17.54.83 [ 133.988705] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 25 00:17:47 EDT 2026 [ 139.663990] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 140.199567] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 155.103643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 155.107105] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.111825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 156.127771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 156.128794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 156.129849] Lustre: Skipped 1 previous similar message [ 156.137905] Lustre: Skipped 1 previous similar message [ 161.247675] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 161.250744] Lustre: Skipped 5 previous similar messages [ 166.367166] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 166.367860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 166.372270] Lustre: Skipped 3 previous similar messages [ 166.442309] Lustre: server umount lustre-MDT0000 complete [ 167.676670] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361100 with bad export cookie 11661705470359525397 [ 167.677791] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 167.682088] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 171.487657] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 171.487897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 171.491641] Lustre: Skipped 2 previous similar messages [ 173.847608] Lustre: server umount lustre-MDT0001 complete [ 181.114464] Lustre: server umount lustre-OST0000 complete [ 188.489542] Lustre: server umount lustre-OST0001 complete [ 190.600437] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 193.368717] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 203.973508] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 204.047522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 204.055809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 204.086038] Lustre: lustre-MDT0000: new disk, initializing [ 204.103210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.108859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 205.310212] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 209.186618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 209.225350] Lustre: 15031:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 209.321501] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 210.534816] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 214.095249] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 215.925633] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 218.227544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.344308] Lustre: lustre-OST0000: new disk, initializing [ 218.345854] Lustre: Skipped 1 previous similar message [ 218.348659] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 218.350869] Lustre: Skipped 2 previous similar messages [ 218.352826] Lustre: 16998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.386885] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 218.390102] Lustre: Skipped 1 previous similar message [ 219.517968] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 219.520914] Lustre: Skipped 1 previous similar message [ 219.523255] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 219.573634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 220.403174] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 224.106401] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 225.393302] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 225.469319] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 226.716960] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 226.779789] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 229.857520] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 229.859502] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 229.860661] LustreError: Skipped 1 previous similar message [ 229.866975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 229.871063] Lustre: Skipped 4 previous similar messages [ 233.484640] Lustre: server umount lustre-OST0000 complete [ 234.976244] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 234.976566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 234.982432] Lustre: Skipped 2 previous similar messages [ 240.790823] Lustre: server umount lustre-MDT0000 complete [ 241.998281] LustreError: 15024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361174 with bad export cookie 11661705470359526860 [ 242.000340] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 242.001024] LustreError: 15024:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 242.110243] Lustre: server umount lustre-MDT0001 complete [ 243.334805] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:19:36 (1782361176) [ 244.104786] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 244.602551] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:19:38 (1782361178) [ 246.705713] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 246.843384] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 246.849404] LustreError: 19406:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 246.867177] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 248.105618] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 250.719985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 252.068131] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 254.290237] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 255.961272] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 258.033142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.918698] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 262.174697] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 275.935733] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 275.938563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 275.943261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 275.945287] Lustre: Skipped 4 previous similar messages [ 281.352202] Lustre: server umount lustre-OST0000 complete [ 286.687833] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 286.691187] LustreError: Skipped 1 previous similar message [ 288.661161] Lustre: server umount lustre-MDT0000 complete [ 289.957643] LustreError: 19388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361222 with bad export cookie 11661705470359527742 [ 289.960731] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 289.961738] LustreError: 19388:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 290.077944] Lustre: server umount lustre-MDT0001 complete [ 293.650141] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 294.638208] Key type lgssc unregistered [ 294.753325] LNet: 22955:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 294.756760] LNetError: 22955:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 294.764263] LNet: Removed LNI 192.168.201.141@tcp [ 295.092119] Key type .llcrypt unregistered [ 295.093326] Key type ._llcrypt unregistered [ 303.029756] Key type ._llcrypt registered [ 303.031338] Key type .llcrypt registered [ 303.068250] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 303.488329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 303.512046] alg: No test for adler32 (adler32-zlib) [ 304.352865] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 304.433101] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 306.007092] Key type lgssc registered [ 306.313527] Lustre: Echo OBD driver; http://www.lustre.org/ [ 308.692400] loop: module loaded [ 308.707983] loop0: detected capacity change from 0 to 204800000 [ 308.776243] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 308.779528] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 308.880182] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 310.513049] loop0: detected capacity change from 0 to 204800000 [ 310.569459] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 310.572656] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 310.575887] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 312.092508] loop0: detected capacity change from 0 to 204800000 [ 313.793411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 313.799968] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 314.890700] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 314.904056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 314.935031] Lustre: lustre-MDT0000: new disk, initializing [ 314.952689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 314.958386] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 316.108016] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 319.351436] loop1: detected capacity change from 0 to 204800000 [ 320.992235] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 321.081754] Lustre: lustre-OST07c6: new disk, initializing [ 321.083485] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 321.085324] Lustre: Skipped 1 previous similar message [ 321.087401] Lustre: 26118:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 321.109781] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 321.790488] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 321.793735] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 321.801867] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 322.786774] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 324.791656] Lustre: 25104:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 324.795425] Lustre: 25104:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 324.798101] Lustre: 25104:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 324.801353] Lustre: 25104:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 324.803512] Lustre: 25104:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 324.805713] Lustre: 25104:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 337.849861] Lustre: 26634:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 337.853372] Lustre: 26634:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 337.856350] Lustre: 26634:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 337.859684] Lustre: 26634:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 337.863608] Lustre: 26634:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 337.867211] Lustre: 26634:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 341.318360] Lustre: 25132:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 341.321222] Lustre: 25132:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 341.323726] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 341.325842] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 341.328028] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 341.330811] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 341.333179] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 341.336760] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 341.340515] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 341.344280] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 341.348258] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 341.352216] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 344.404053] Lustre: 26634:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 344.407014] Lustre: 26634:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 344.409526] Lustre: 26634:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 344.411794] Lustre: 26634:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 344.414168] Lustre: 26634:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 344.416357] Lustre: 26634:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 344.418654] Lustre: 26634:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 344.421119] Lustre: 26634:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 344.423553] Lustre: 26634:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 344.425711] Lustre: 26634:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 344.428181] Lustre: 26634:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 344.430708] Lustre: 26634:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 350.873341] Lustre: 26634:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 350.876627] Lustre: 26634:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 350.879907] Lustre: 26634:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 350.883429] Lustre: 26634:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 350.886334] Lustre: 26634:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 350.889787] Lustre: 26634:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 350.892197] Lustre: 26634:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 350.895601] Lustre: 26634:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 350.898074] Lustre: 26634:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 350.901124] Lustre: 26634:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 350.903680] Lustre: 26634:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 350.907032] Lustre: 26634:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 360.072149] Lustre: 25132:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 360.075190] Lustre: 25132:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 360.077466] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 360.079764] Lustre: 25132:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 360.082144] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 360.085198] Lustre: 25132:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 360.089054] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 360.092522] Lustre: 25132:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 360.094579] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 360.096647] Lustre: 25132:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 360.098843] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 360.100995] Lustre: 25132:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 403.937085] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 403.937146] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 403.946672] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 406.924407] Lustre: server umount lustre-OST07c6 complete [ 410.135866] Lustre: server umount lustre-MDT0000 complete [ 415.624817] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 416.631193] Key type lgssc unregistered [ 416.756363] LNet: 28695:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 416.759912] LNetError: 28695:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 416.770446] LNet: Removed LNI 192.168.201.141@tcp [ 417.106978] Key type .llcrypt unregistered [ 417.108767] Key type ._llcrypt unregistered [ 424.311023] Key type ._llcrypt registered [ 424.313017] Key type .llcrypt registered [ 424.350462] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 424.706283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 424.750248] alg: No test for adler32 (adler32-zlib) [ 425.606820] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 425.693410] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 427.271128] Key type lgssc registered [ 427.621761] Lustre: Echo OBD driver; http://www.lustre.org/ [ 430.160097] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 430.167481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 431.321523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 432.618546] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 435.370069] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 435.495935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 436.835908] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 439.119196] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 440.885391] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 443.121755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 443.244592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 445.251051] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 447.649321] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 456.548946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 465.888624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.890260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.892759] LustreError: Skipped 2 previous similar messages [ 465.893408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.898852] Lustre: Skipped 1 previous similar message [ 466.912993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 466.914981] Lustre: Skipped 2 previous similar messages [ 468.883478] Lustre: server umount lustre-OST0000 complete [ 471.519852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 471.522298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.526480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.439597] Lustre: server umount lustre-MDT0000 complete [ 477.151932] LustreError: 29941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.156759] LustreError: 29941:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 477.641366] LustreError: 29922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361410 with bad export cookie 17035868812083085161 [ 477.644929] LustreError: 29922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 477.644984] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 477.757689] Lustre: server umount lustre-MDT0001 complete [ 481.284487] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 482.383194] Key type lgssc unregistered [ 482.527405] LNet: 33694:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 482.530360] LNetError: 33694:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 482.542334] LNet: Removed LNI 192.168.201.141@tcp [ 482.873343] Key type .llcrypt unregistered [ 482.874970] Key type ._llcrypt unregistered [ 490.270358] Key type ._llcrypt registered [ 490.272318] Key type .llcrypt registered [ 490.318334] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 490.677712] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 490.722290] alg: No test for adler32 (adler32-zlib) [ 491.561624] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 491.642564] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 493.215098] Key type lgssc registered [ 493.550980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 495.950556] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 495.955839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 497.100878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.368423] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 501.160262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 501.288649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 502.676298] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 505.027532] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.798138] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 509.034658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.151720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 511.090195] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 513.433585] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 522.470608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 531.936420] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 531.939059] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.943309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.961677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 532.962797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 532.965420] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 532.967037] Lustre: Skipped 1 previous similar message [ 538.081038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 538.084016] Lustre: Skipped 1 previous similar message [ 542.175248] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 542.229127] Lustre: server umount lustre-OST0000 complete [ 547.295919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 547.298435] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 547.302474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 549.595622] Lustre: server umount lustre-MDT0000 complete [ 550.847194] LustreError: 34921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361483 with bad export cookie 8897163011037376706 [ 550.849771] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 550.853118] LustreError: 34921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 550.972027] Lustre: server umount lustre-MDT0001 complete [ 554.504584] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 555.508612] Key type lgssc unregistered [ 555.638385] LNet: 38492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 555.640775] LNetError: 38492:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 555.648308] LNet: Removed LNI 192.168.201.141@tcp [ 555.952340] Key type .llcrypt unregistered [ 555.954324] Key type ._llcrypt unregistered [ 564.886292] Key type ._llcrypt registered [ 564.887509] Key type .llcrypt registered [ 564.927271] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 565.263148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 565.314218] alg: No test for adler32 (adler32-zlib) [ 566.161022] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 566.249400] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 567.831130] Key type lgssc registered [ 568.204524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 570.757268] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 570.765447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 571.912608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 573.209511] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 575.983111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 576.109108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 577.453828] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 579.768212] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 581.555818] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 583.807212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.926069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 585.875648] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 588.273374] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 596.199269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 601.568864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.570678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 601.571130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 601.573363] Lustre: Skipped 2 previous similar messages [ 604.085923] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 604.087959] Lustre: Skipped 2 previous similar messages [ 606.689239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 606.692575] Lustre: Skipped 2 previous similar messages [ 607.379964] Lustre: server umount lustre-MDT0000 complete [ 608.623213] LustreError: 39867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361541 with bad export cookie 10691846083624142056 [ 608.626615] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 608.629187] LustreError: 39867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 609.198888] LustreError: 39883:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 609.204980] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 609.206219] LustreError: 39883:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 611.808086] LustreError: 42375:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.808277] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 613.856257] LustreError: 41319:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 613.856606] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 613.863283] Lustre: Skipped 1 previous similar message [ 614.763303] Lustre: server umount lustre-MDT0001 complete [ 632.621776] Lustre: server umount lustre-OST0000 complete [ 638.028631] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 639.023567] Key type lgssc unregistered [ 639.153364] LNet: 43768:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 639.156353] LNetError: 43768:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 639.168247] LNet: Removed LNI 192.168.201.141@tcp [ 639.473350] Key type .llcrypt unregistered [ 639.474428] Key type ._llcrypt unregistered [ 648.418028] Key type ._llcrypt registered [ 648.419148] Key type .llcrypt registered [ 648.458806] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 648.837512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.845631] alg: No test for adler32 (adler32-zlib) [ 649.702179] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 649.791526] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 651.375203] Key type lgssc registered [ 651.755165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.364218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 654.369742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 655.516808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 656.928378] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 659.717500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 659.843691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 661.202167] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 663.478975] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 665.154518] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 667.330056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 667.442889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 669.287890] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 671.643181] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 681.766490] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 685.023821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 685.027643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 685.033536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 687.073335] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 687.074776] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 687.075298] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 690.709149] Lustre: server umount lustre-OST0000 complete [ 703.309312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.310777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 703.313498] Lustre: Skipped 1 previous similar message [ 703.315312] Lustre: Skipped 2 previous similar messages [ 708.576168] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 708.578158] Lustre: Skipped 2 previous similar messages [ 709.590731] Lustre: server umount lustre-MDT0000 complete [ 710.816062] LustreError: 45144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361643 with bad export cookie 18347169966008172733 [ 710.819160] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 710.820914] LustreError: 45144:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 716.976171] Lustre: server umount lustre-MDT0001 complete [ 720.520228] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 721.522874] Key type lgssc unregistered [ 721.647434] LNet: 48890:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 721.651349] LNetError: 48890:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 721.659368] LNet: Removed LNI 192.168.201.141@tcp [ 721.996040] Key type .llcrypt unregistered [ 721.997102] Key type ._llcrypt unregistered [ 730.683960] Key type ._llcrypt registered [ 730.685077] Key type .llcrypt registered [ 730.725592] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 731.184691] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 731.202567] alg: No test for adler32 (adler32-zlib) [ 732.056077] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 732.141894] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 733.719157] Key type lgssc registered [ 734.102861] Lustre: Echo OBD driver; http://www.lustre.org/ [ 736.777280] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 736.783565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 737.931388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 739.216809] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 742.051738] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 742.171520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 743.487644] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 745.798443] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 747.512600] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 749.658160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 749.769489] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 751.712444] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 754.150195] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 762.087048] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 766.850780] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 767.909440] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 778.619451] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-25 0h28m31s [ 779.101104] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 780.212471] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 781.230875] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 782.818393] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 782.820713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 782.822916] Lustre: Skipped 2 previous similar messages [ 782.824730] Lustre: Skipped 1 previous similar message [ 787.374221] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 787.376844] Lustre: Skipped 4 previous similar messages [ 792.430394] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 792.432836] Lustre: Skipped 3 previous similar messages [ 794.079116] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 794.141067] Lustre: server umount lustre-MDT0000 complete [ 795.105745] LustreError: 50279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 795.111073] LustreError: 50279:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 795.266394] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 796.128985] LustreError: 50278:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 796.134498] LustreError: 50278:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 797.410567] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 797.453697] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 797.540243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 797.563499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 798.565308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 798.863376] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 799.989159] Lustre: DEBUG MARKER: Wait for df (36368) ... [ 800.502102] Lustre: DEBUG MARKER: done [ 802.783966] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 802.787036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 802.791097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 803.542578] Lustre: server umount lustre-OST0000 complete [ 808.928747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 808.928907] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 810.900473] Lustre: server umount lustre-MDT0000 complete [ 812.091152] LustreError: 50263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361745 with bad export cookie 11581098130090719890 [ 812.094275] LustreError: 50263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 812.095418] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 822.294558] Lustre: server umount lustre-MDT0001 complete [ 825.803284] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 826.834345] Key type lgssc unregistered [ 826.969343] LNet: 55217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 826.972700] LNetError: 55217:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 826.981341] LNet: Removed LNI 192.168.201.141@tcp [ 827.300214] Key type .llcrypt unregistered [ 827.301283] Key type ._llcrypt unregistered [ 841.377956] Key type ._llcrypt registered [ 841.379907] Key type .llcrypt registered [ 841.417493] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 841.764510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 841.794563] alg: No test for adler32 (adler32-zlib) [ 842.650554] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 842.733487] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 844.311154] Key type lgssc registered [ 844.688363] Lustre: Echo OBD driver; http://www.lustre.org/ [ 847.302620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 847.310882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 848.395696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 848.403389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 848.461422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 849.834779] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 852.714935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 852.747898] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 852.759959] Lustre: 57401:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 852.859471] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 854.218383] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 856.593141] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 858.393183] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 860.582557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 860.650615] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 860.718730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 861.776117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 862.603792] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 864.968348] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 868.279681] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-25 0h30m01s [ 868.790589] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 871.080699] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 871.857428] Lustre: DEBUG MARKER: Creating a test file and stat it [ 871.871268] LustreError: 57408:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 892.347861] Lustre: lustre-MDT0000: Client 21bbc592-629d-47a6-abbf-170021190fb4 (at 192.168.201.41@tcp) reconnecting [ 911.911107] LustreError: 57408:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 912.666736] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 913.495425] Lustre: DEBUG MARKER: done [ 917.986732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 917.986840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 917.988743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 917.989550] LustreError: Skipped 2 previous similar messages [ 917.997380] Lustre: Skipped 1 previous similar message [ 921.169229] Lustre: server umount lustre-OST0000 complete [ 923.104762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 923.105470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 923.108434] Lustre: Skipped 1 previous similar message [ 923.111565] Lustre: Skipped 2 previous similar messages [ 924.128527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 924.130601] Lustre: Skipped 1 previous similar message [ 924.506108] Lustre: server umount lustre-MDT0000 complete [ 925.728075] LustreError: 57394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361858 with bad export cookie 12905289462512278962 [ 925.731254] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 925.732911] LustreError: 57394:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 927.853554] Lustre: server umount lustre-MDT0001 complete [ 931.320822] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 932.344945] Key type lgssc unregistered [ 932.465351] LNet: 61724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.467991] LNetError: 61724:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.475289] LNet: Removed LNI 192.168.201.141@tcp [ 932.786770] Key type .llcrypt unregistered [ 932.788032] Key type ._llcrypt unregistered [ 948.036808] Key type ._llcrypt registered [ 948.037930] Key type .llcrypt registered [ 948.074853] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 948.528326] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.538258] alg: No test for adler32 (adler32-zlib) [ 949.389145] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 949.480400] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 951.063148] Key type lgssc registered [ 951.397143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 952.444757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 952.450535] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 953.535411] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 953.541645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 953.594951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.924331] Lustre: Failing over lustre-MDT0000 [ 954.018649] Lustre: server umount lustre-MDT0000 complete [ 956.121332] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 00:31:29 (1782361889) [ 958.251560] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.381778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 959.679870] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 962.404758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 962.436410] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 962.446878] Lustre: 64497:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 962.546511] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 963.888996] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 966.197139] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 967.927638] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 970.064264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 970.138761] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 970.199870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.171022] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 974.710606] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 976.400579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 982.702294] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 986.593787] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 986.594890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 986.598297] Lustre: Skipped 2 previous similar messages [ 986.600119] Lustre: Skipped 1 previous similar message [ 990.363129] Lustre: server umount lustre-MDT0000 complete [ 991.546191] LustreError: 64489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361924 with bad export cookie 8941642461427969055 [ 991.549254] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 991.550150] LustreError: 64489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 991.654536] Lustre: server umount lustre-MDT0001 complete [ 992.717928] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 994.176202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 996.859400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1001.951666] LustreError: 63419:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000cb077aef x1868941892812800/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1002.066597] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1002.094992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1002.132063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1003.390757] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1006.144164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1006.179434] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1007.263078] Lustre: 63420:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782361924/real 1782361924] req@000000006a58db1d x1868941892812416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782361940 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1007.570626] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1009.862300] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1011.607140] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1011.684632] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1012.703154] Lustre: 63420:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782361929/real 1782361929] req@000000004b3e9378 x1868941892812544/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782361945 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1016.800950] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1016.803076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1016.803990] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1016.803995] Lustre: Skipped 1 previous similar message [ 1016.810373] Lustre: Skipped 1 previous similar message [ 1017.823229] Lustre: 63423:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782361934/real 1782361934] req@00000000c5a4a66a x1868941892812672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782361950 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1019.417829] Lustre: server umount lustre-MDT0000 complete [ 1020.623246] LustreError: 67071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361953 with bad export cookie 8941642461427971323 [ 1020.626458] LustreError: 67071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1020.626565] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1020.727893] Lustre: server umount lustre-MDT0001 complete [ 1022.768441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1025.768426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1031.071703] LustreError: 63419:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000007076651c x1868941892831488/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1031.179685] LustreError: 68994:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1031.184742] LustreError: 68994:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 1031.202157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1031.205103] Lustre: Skipped 1 previous similar message [ 1031.229322] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1032.491035] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1035.205302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.316820] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1035.321548] Lustre: Skipped 1 previous similar message [ 1036.389419] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1036.648436] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1037.407114] Lustre: 63423:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782361954/real 1782361954] req@000000001ef5480e x1868941892831104/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782361970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1039.049068] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1040.866897] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1041.471139] Lustre: 63420:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782361958/real 1782361958] req@000000001abc9946 x1868941892831232/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782361974 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1045.472177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1045.474528] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1045.478933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1045.480663] Lustre: Skipped 2 previous similar messages [ 1048.665920] Lustre: server umount lustre-MDT0000 complete [ 1049.907184] LustreError: 67071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782361982 with bad export cookie 8941642461427972702 [ 1049.909461] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1049.910641] LustreError: 67071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1050.029033] Lustre: server umount lustre-MDT0001 complete [ 1052.159565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1053.310603] Lustre: DEBUG MARKER: files compared the same [ 1063.260537] Lustre: server umount lustre-OST0000 complete [ 1068.438187] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1069.441654] Key type lgssc unregistered [ 1069.577331] LNet: 74212:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1069.579425] LNetError: 74212:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1069.585401] LNet: Removed LNI 192.168.201.141@tcp [ 1069.899052] Key type .llcrypt unregistered [ 1069.900106] Key type ._llcrypt unregistered [ 1078.707608] Key type ._llcrypt registered [ 1078.708627] Key type .llcrypt registered [ 1078.741275] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1079.042813] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1079.077023] alg: No test for adler32 (adler32-zlib) [ 1079.931495] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1080.016919] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1081.591147] Key type lgssc registered [ 1081.948038] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1084.382466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1084.387955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1085.519519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1086.847305] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug +malloc all [ 1089.560704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.648974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1090.893790] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug +malloc all [ 1093.187913] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1094.888587] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1097.198742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1097.293419] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1099.186484] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug +malloc all [ 1101.496804] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1111.651315] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1115.103564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1115.106885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1115.111794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.152462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1117.152842] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1117.164872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1120.868398] Lustre: server umount lustre-OST0000 complete [ 1125.855759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1125.860321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1125.868567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1125.872668] Lustre: Skipped 1 previous similar message [ 1128.809597] Lustre: server umount lustre-MDT0000 complete [ 1130.589202] LustreError: 75585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362063 with bad export cookie 744711307356854837 [ 1130.591612] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1130.596780] LustreError: 75585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1130.746718] Lustre: server umount lustre-MDT0001 complete [ 1135.564863] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1136.757585] Key type lgssc unregistered [ 1136.916645] LNet: 79154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1136.921168] LNetError: 79154:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1136.931508] LNet: Removed LNI 192.168.201.141@tcp [ 1137.280969] Key type .llcrypt unregistered [ 1137.282665] Key type ._llcrypt unregistered [ 1146.756659] Key type ._llcrypt registered [ 1146.757703] Key type .llcrypt registered [ 1146.793968] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1147.182851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1147.198471] alg: No test for adler32 (adler32-zlib) [ 1148.046422] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1148.131081] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1149.711154] Key type lgssc registered [ 1150.048411] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1152.543941] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1152.549757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1167.967397] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1173.087337] LustreError: 80520:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 1199.711322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1202.145200] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1205.305868] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1208.704252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.740076] Lustre: *** cfs_fail_loc=706, val=0*** [ 1208.868856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1208.895340] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1210.347063] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1213.686212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1213.843515] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1215.194792] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1218.060624] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1219.701374] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1224.161082] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1224.162073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1224.165745] LustreError: Skipped 1 previous similar message [ 1224.174199] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1224.181661] Lustre: Skipped 1 previous similar message [ 1225.185242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1225.187026] Lustre: Skipped 1 previous similar message [ 1226.711915] Lustre: server umount lustre-OST0000 complete [ 1229.280103] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1229.282578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1229.285530] Lustre: Skipped 1 previous similar message [ 1229.287176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1230.306040] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1234.397535] Lustre: server umount lustre-MDT0000 complete [ 1235.424318] LustreError: 83383:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1235.428599] LustreError: 83383:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1235.714696] LustreError: 80526:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362168 with bad export cookie 2407655832571627587 [ 1235.719631] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1235.843971] Lustre: server umount lustre-MDT0001 complete [ 1239.765982] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1240.859147] Key type lgssc unregistered [ 1240.984494] LNet: 84133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1240.988813] LNetError: 84133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1240.997561] LNet: Removed LNI 192.168.201.141@tcp [ 1241.342499] Key type .llcrypt unregistered [ 1241.343588] Key type ._llcrypt unregistered [ 1251.283687] Key type ._llcrypt registered [ 1251.284869] Key type .llcrypt registered [ 1251.334407] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1251.739504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1251.794153] alg: No test for adler32 (adler32-zlib) [ 1252.652773] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1252.745312] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1254.327171] Key type lgssc registered [ 1254.872409] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1257.615028] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1257.621486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1260.141306] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1262.757899] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1262.952613] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1264.241948] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1267.185713] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1267.290898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.256301] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1272.263613] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1272.271846] Lustre: MGS: Client 8c1103fc-70b5-4135-825b-3a60d0d55f27 (at 0@lo) reconnecting [ 1272.286115] LustreError: 86226:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1272.379122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1272.406045] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1273.828437] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1282.528909] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1282.532477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1282.539448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1283.558106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1283.558311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.571661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1287.700298] Lustre: server umount lustre-OST0000 complete [ 1293.792843] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1293.792904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1293.793532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1293.793537] Lustre: Skipped 1 previous similar message [ 1293.796527] Lustre: Skipped 1 previous similar message [ 1295.303958] Lustre: server umount lustre-MDT0000 complete [ 1296.973606] Lustre: server umount lustre-MDT0001 complete [ 1298.353962] Lustre: server umount MGS complete [ 1302.447922] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 00:37:15 (1782362235) [ 1311.834839] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 1315.121797] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1326.723632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1328.067173] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1331.928057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1332.013641] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1332.031289] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1332.069918] Lustre: lustre-MDT0001: new disk, initializing [ 1332.094845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1332.103944] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1332.106402] Lustre: 93374:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1332.109199] LustreError: 93374:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1333.388859] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1337.626654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1337.734107] Lustre: lustre-OST0000: new disk, initializing [ 1337.736101] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1337.738648] Lustre: 94140:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1337.761002] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.783324] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1343.896596] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1345.056532] Lustre: 94849:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000f2ec1821 x1868942210532864/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782362293 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1347.050055] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1347.056105] Lustre: MGS: Client ec8605eb-0635-49af-bdea-859321ba2170 (at 0@lo) reconnecting [ 1347.075727] Lustre: 95186:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1347.088412] LustreError: 93359:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1347.091875] LustreError: 93359:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1347.099267] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1347.142302] Lustre: lustre-MDT0000: new disk, initializing [ 1347.186264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.195430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1348.256351] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1348.256496] Lustre: 95202:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000ae937fc1 x1868942210538496/t0(0) o700->lustre-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782362297 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1348.266865] Lustre: 95202:0:(client.c:1638:after_reply()) Skipped 1 previous similar message [ 1348.759775] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1353.122562] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1353.126987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1353.315421] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1373.664001] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1373.667886] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.674418] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1373.676588] Lustre: Skipped 1 previous similar message [ 1382.374206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1382.378306] Lustre: Skipped 4 previous similar messages [ 1386.975223] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1387.056688] Lustre: server umount lustre-OST0000 complete [ 1392.610133] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1392.610184] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1392.611923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1392.617017] Lustre: Skipped 2 previous similar messages [ 1392.621516] LustreError: Skipped 1 previous similar message [ 1394.501736] Lustre: server umount lustre-MDT0000 complete [ 1401.831309] Lustre: server umount lustre-MDT0001 complete [ 1403.491496] Lustre: server umount MGS complete [ 1407.853149] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 00:39:01 (1782362341) [ 1414.491099] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1415.468670] Key type lgssc unregistered [ 1415.593369] LNet: 97707:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1415.596744] LNetError: 97707:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1415.611362] LNet: Removed LNI 192.168.201.141@tcp [ 1415.907049] Key type .llcrypt unregistered [ 1415.908237] Key type ._llcrypt unregistered [ 1423.261524] Key type ._llcrypt registered [ 1423.262766] Key type .llcrypt registered [ 1423.295615] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1423.685217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1423.695219] alg: No test for adler32 (adler32-zlib) [ 1424.545907] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1424.631888] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1426.207161] Key type lgssc registered [ 1426.584826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1429.296039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1429.301483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1430.441376] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.787211] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1434.641363] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1434.760303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1436.041814] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1438.526198] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1440.317181] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1442.861219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1442.983407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1445.034040] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1447.318666] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1456.295853] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1460.704283] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1460.706491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1460.710189] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1461.728421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1461.730954] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1461.731205] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1461.737730] Lustre: Skipped 1 previous similar message [ 1462.743467] Lustre: server umount lustre-OST0000 complete [ 1465.824314] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1465.826512] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1465.830300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.151958] Lustre: server umount lustre-MDT0000 complete [ 1466.849836] LustreError: 101505:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1466.859570] LustreError: 101505:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1467.645146] LustreError: 99640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362400 with bad export cookie 5697167747087994521 [ 1467.648000] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1467.653036] LustreError: 99640:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1467.763353] Lustre: server umount lustre-MDT0001 complete [ 1471.180696] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1472.319393] Key type lgssc unregistered [ 1472.484830] LNet: 102456:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1472.489439] LNetError: 102456:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1472.499706] LNet: Removed LNI 192.168.201.141@tcp [ 1472.870581] Key type .llcrypt unregistered [ 1472.871927] Key type ._llcrypt unregistered [ 1479.059171] Key type ._llcrypt registered [ 1479.060216] Key type .llcrypt registered [ 1479.094924] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1479.528775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1479.537350] alg: No test for adler32 (adler32-zlib) [ 1480.385349] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1480.480849] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1482.055215] Key type lgssc registered [ 1482.478733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1484.707776] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1484.707892] LustreError: 103516:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1484.710992] Lustre: Skipped 1 previous similar message [ 1484.716032] LustreError: 103517:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1484.719433] LustreError: 103516:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 1484.722830] LustreError: 103516:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1484.727100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1484.728118] LustreError: 103516:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1485.903756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1488.597679] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1488.683994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1490.040805] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1491.634562] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing lsmod [ 1492.565468] LustreError: 104720:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1492.570791] LustreError: 104721:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1492.574823] LustreError: 104720:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 1492.576986] LustreError: 104720:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1492.580542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1492.581097] LustreError: 104720:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1492.713685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1493.665180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1493.667445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1493.669465] Lustre: Skipped 2 previous similar messages [ 1493.671475] Lustre: Skipped 1 previous similar message [ 1498.784861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1498.787750] Lustre: Skipped 1 previous similar message [ 1499.803670] Lustre: server umount lustre-MDT0000 complete [ 1501.053989] LustreError: 104759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362433 with bad export cookie 13837763317033519828 [ 1501.056690] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1501.057076] LustreError: 104759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1502.879302] LustreError: 103984:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009718e6c0 x1868942449589632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1502.931656] Lustre: server umount lustre-MDT0001 complete [ 1514.519665] Lustre: server umount lustre-OST0000 complete [ 1517.303206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1517.456370] LustreError: 105785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.462415] LustreError: 105785:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1517.479827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1518.774141] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1521.483869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1521.602311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1522.921700] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1525.236318] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1527.049056] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1529.450676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1531.286304] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1533.551956] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1542.882878] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1547.232748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1547.235180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.239648] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1548.256743] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1548.258764] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.436328] Lustre: server umount lustre-OST0000 complete [ 1553.376992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.377654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1553.381218] Lustre: Skipped 1 previous similar message [ 1553.384480] Lustre: Skipped 3 previous similar messages [ 1558.496205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1558.498786] Lustre: Skipped 1 previous similar message [ 1558.743730] Lustre: server umount lustre-MDT0000 complete [ 1559.925223] LustreError: 105768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362492 with bad export cookie 13837763317033520500 [ 1559.928491] LustreError: 105768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1559.928771] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1560.031708] Lustre: server umount lustre-MDT0001 complete [ 1563.371916] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1564.324547] Key type lgssc unregistered [ 1564.457282] LNet: 109336:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1564.459580] LNetError: 109336:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1564.467325] LNet: Removed LNI 192.168.201.141@tcp [ 1564.769407] Key type .llcrypt unregistered [ 1564.770716] Key type ._llcrypt unregistered [ 1573.479982] Key type ._llcrypt registered [ 1573.481072] Key type .llcrypt registered [ 1573.518033] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1573.837801] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1573.870894] alg: No test for adler32 (adler32-zlib) [ 1574.723522] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1574.812581] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1576.391158] Key type lgssc registered [ 1576.749152] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1579.232825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1579.240148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1580.380234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.655332] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1584.476395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.595412] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1585.904389] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1588.131336] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1589.898186] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1592.117721] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1592.229133] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1594.120894] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1596.435016] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1605.545580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1609.051936] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1609.803731] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1610.720374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1610.721207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1610.721890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1610.728255] Lustre: Skipped 1 previous similar message [ 1615.840978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1615.843960] Lustre: Skipped 2 previous similar messages [ 1616.787064] Lustre: server umount lustre-OST0000 complete [ 1620.447818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1620.451164] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.457778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1626.079948] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1626.081792] Lustre: Skipped 2 previous similar messages [ 1631.201564] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1631.204178] Lustre: Skipped 1 previous similar message [ 1632.223192] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1632.322282] Lustre: server umount lustre-MDT0000 complete [ 1633.744062] LustreError: 110708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362566 with bad export cookie 18045995596177423693 [ 1633.747167] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1633.748552] LustreError: 110708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1633.863626] Lustre: server umount lustre-MDT0001 complete [ 1637.381158] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1638.413439] Key type lgssc unregistered [ 1638.557585] LNet: 114384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1638.561678] LNetError: 114384:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1638.576437] LNet: Removed LNI 192.168.201.141@tcp [ 1638.958576] Key type .llcrypt unregistered [ 1638.960512] Key type ._llcrypt unregistered [ 1645.798884] Key type ._llcrypt registered [ 1645.799923] Key type .llcrypt registered [ 1645.834962] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1646.252617] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1646.259298] alg: No test for adler32 (adler32-zlib) [ 1647.105175] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1647.186795] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1648.759205] Key type lgssc registered [ 1649.088112] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1652.044784] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1652.051581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1653.225172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1654.579351] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1657.929035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1658.075628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1659.684079] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1662.317951] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1664.484677] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1666.894437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1667.017753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1668.947934] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1671.341498] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1678.311852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1688.545301] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1688.546943] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.547823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1688.553165] Lustre: Skipped 1 previous similar message [ 1690.894550] Lustre: server umount lustre-OST0000 complete [ 1693.665746] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1693.666534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1693.670098] Lustre: Skipped 1 previous similar message [ 1693.672031] Lustre: Skipped 1 previous similar message [ 1698.200747] Lustre: server umount lustre-MDT0000 complete [ 1698.784835] LustreError: 115581:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1698.792172] LustreError: 115581:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1699.393343] LustreError: 115562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362632 with bad export cookie 3866132886458990452 [ 1699.397356] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.399488] LustreError: 115562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1699.519813] Lustre: server umount lustre-MDT0001 complete [ 1702.936617] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1703.991529] Key type lgssc unregistered [ 1704.121428] LNet: 119133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1704.125741] LNetError: 119133:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1704.136390] LNet: Removed LNI 192.168.201.141@tcp [ 1704.482292] Key type .llcrypt unregistered [ 1704.483411] Key type ._llcrypt unregistered [ 1713.614551] Key type ._llcrypt registered [ 1713.615615] Key type .llcrypt registered [ 1713.652496] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1714.123340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.129952] alg: No test for adler32 (adler32-zlib) [ 1715.006837] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1715.121985] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1716.711224] Key type lgssc registered [ 1717.097805] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1719.620816] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1719.626486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1720.766038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.070229] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1724.854366] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1724.994635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1726.308554] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1728.444355] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1730.114973] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1732.301251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1732.409386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1734.283923] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1736.591079] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1745.765717] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1749.255858] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1750.465210] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1750.467222] Lustre: Skipped 2 previous similar messages [ 1751.844772] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1751.847164] Lustre: Skipped 2 previous similar messages [ 1759.736424] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1759.745383] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1761.767044] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.41@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1761.770301] Lustre: Skipped 2 previous similar messages [ 1768.443853] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.41@tcp 192.168.0.[2,10]@tcp [ 1769.504770] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.41/24@tcp in log lustre-MDT0000 [ 1769.507759] Lustre: Skipped 2 previous similar messages [ 1775.607042] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.41/24@tcp [ 1775.609689] Lustre: Skipped 1 previous similar message [ 1780.703744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1780.705924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1780.709515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.729833] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1781.730856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1781.733473] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1783.377415] Lustre: server umount lustre-OST0000 complete [ 1786.335955] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1786.338043] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.341788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1786.343475] Lustre: Skipped 1 previous similar message [ 1790.746468] Lustre: server umount lustre-MDT0000 complete [ 1791.968818] LustreError: 120524:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1791.974323] LustreError: 120524:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1791.982153] LustreError: 120549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362724 with bad export cookie 17250838075049568673 [ 1791.984750] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1791.985167] LustreError: 120549:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1792.096944] Lustre: server umount lustre-MDT0001 complete [ 1795.452124] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1796.497345] Key type lgssc unregistered [ 1796.616369] LNet: 125860:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1796.618786] LNetError: 125860:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1796.628277] LNet: Removed LNI 192.168.201.141@tcp [ 1796.949989] Key type .llcrypt unregistered [ 1796.951795] Key type ._llcrypt unregistered [ 1807.260758] Key type ._llcrypt registered [ 1807.261866] Key type .llcrypt registered [ 1807.298307] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1807.612727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1807.649070] alg: No test for adler32 (adler32-zlib) [ 1808.496867] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1808.579524] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1810.159100] Key type lgssc registered [ 1810.512604] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1812.962734] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1812.968682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1814.111333] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1814.112359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.119759] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.41/24@tcp [ 1815.431475] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1818.217596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.322116] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1818.330281] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.41/24@tcp [ 1818.344240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1819.725159] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1822.078484] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1823.846782] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1826.128954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1826.261071] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1828.300174] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1830.775370] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1839.590967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1844.705495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1844.706025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1844.708821] Lustre: Skipped 1 previous similar message [ 1844.712075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1849.824785] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.827326] Lustre: Skipped 2 previous similar messages [ 1850.774891] Lustre: server umount lustre-OST0000 complete [ 1854.432099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1854.434903] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1854.439581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1858.206908] Lustre: server umount lustre-MDT0000 complete [ 1859.513479] LustreError: 127377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362792 with bad export cookie 15085841974545176879 [ 1859.516954] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1859.517168] LustreError: 127377:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1859.639499] Lustre: server umount lustre-MDT0001 complete [ 1863.094280] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 1864.116051] Key type lgssc unregistered [ 1864.242336] LNet: 130999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1864.244786] LNetError: 130999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1864.253309] LNet: Removed LNI 192.168.201.141@tcp [ 1864.555037] Key type .llcrypt unregistered [ 1864.556124] Key type ._llcrypt unregistered [ 1873.012833] Key type ._llcrypt registered [ 1873.014295] Key type .llcrypt registered [ 1873.057047] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 1879.225203] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 1879.629351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1879.636114] alg: No test for adler32 (adler32-zlib) [ 1880.475278] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 1880.560213] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 1882.135151] Key type lgssc registered [ 1882.502841] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1893.302266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1893.309919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1894.390743] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1894.401443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1894.438247] Lustre: lustre-MDT0000: new disk, initializing [ 1894.459268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1894.465449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1895.786011] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1899.706837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1899.745627] Lustre: 134371:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1899.770093] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1899.772899] Lustre: Skipped 1 previous similar message [ 1899.809096] Lustre: lustre-MDT0001: new disk, initializing [ 1899.839235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1899.850095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1899.854649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1901.104641] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1904.686724] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1906.442928] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1908.657172] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1908.755788] Lustre: lustre-OST0000: new disk, initializing [ 1908.757430] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1908.760027] Lustre: 136336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1908.788166] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1910.600128] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1910.651186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1910.655857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1910.688136] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1914.086646] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1915.377639] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1915.444950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1916.649680] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1916.713554] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1919.372304] Lustre: server umount lustre-OST0000 complete [ 1920.480740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1920.483241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.486956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1920.992295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1920.998571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.112301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1926.115242] Lustre: Skipped 1 previous similar message [ 1930.239457] Lustre: server umount lustre-MDT0000 complete [ 1931.563927] LustreError: 134362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362864 with bad export cookie 7388311301927651646 [ 1931.567903] LustreError: 134362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1931.568670] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1942.056544] Lustre: server umount lustre-MDT0001 complete [ 1945.220328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.372361] LustreError: 138554:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1945.377663] LustreError: 138554:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1945.394614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1946.734444] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1949.441232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.558749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.998355] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1953.195270] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1955.069609] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1957.419034] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1959.337772] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1961.852624] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1963.322756] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1971.562401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1972.840955] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1972.907994] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1976.149554] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1976.206281] Lustre: lustre-OST0001: new disk, initializing [ 1976.207792] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1976.209943] Lustre: 141978:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1976.238767] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1976.240974] Lustre: Skipped 1 previous similar message [ 1978.125693] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 1981.244148] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1986.561316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1986.564191] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1986.590949] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1987.446315] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 6 sec [ 1988.802952] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1988.872343] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1990.692406] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90cd080ec800.ost_server_uuid 50 [ 1991.151911] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90cd080ec800.ost_server_uuid in FULL state after 0 sec [ 1996.776617] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1996.778985] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1996.780487] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1996.782842] Lustre: Skipped 1 previous similar message [ 1996.785564] Lustre: Skipped 1 previous similar message [ 1998.974161] Lustre: server umount lustre-OST0001 complete [ 2000.864653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2000.867852] LustreError: Skipped 1 previous similar message [ 2000.869595] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2000.874589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2001.896122] LustreError: 140374:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2001.907525] LustreError: 140374:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2006.283712] Lustre: server umount lustre-OST0000 complete [ 2011.104133] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2011.106920] LustreError: Skipped 1 previous similar message [ 2011.108170] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.113594] Lustre: Skipped 1 previous similar message [ 2011.116334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2011.118849] Lustre: Skipped 2 previous similar messages [ 2013.723275] Lustre: server umount lustre-MDT0000 complete [ 2014.929140] LustreError: 138535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782362947 with bad export cookie 7388311301927652514 [ 2014.932441] LustreError: 138535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2014.933565] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.051929] Lustre: server umount lustre-MDT0001 complete [ 2021.352681] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2022.384568] Key type lgssc unregistered [ 2022.510397] LNet: 145017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2022.512670] LNetError: 145017:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2022.522376] LNet: Removed LNI 192.168.201.141@tcp [ 2022.829027] Key type .llcrypt unregistered [ 2022.829978] Key type ._llcrypt unregistered [ 2035.273578] Key type ._llcrypt registered [ 2035.274648] Key type .llcrypt registered [ 2035.310489] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2041.197971] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2041.511290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2041.549791] alg: No test for adler32 (adler32-zlib) [ 2042.390996] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 2042.475241] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2044.047103] Key type lgssc registered [ 2044.377322] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2054.837829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2054.842358] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2055.926492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2055.935519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2055.972149] Lustre: lustre-MDT0000: new disk, initializing [ 2055.992744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2056.000483] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2057.302734] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2061.245771] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2061.287227] Lustre: 149143:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2061.311805] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2061.313433] Lustre: Skipped 1 previous similar message [ 2061.346759] Lustre: lustre-MDT0001: new disk, initializing [ 2061.380506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2061.392030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2061.395972] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2062.667707] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2066.213133] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2067.937118] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2070.080857] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2070.183172] Lustre: lustre-OST0000: new disk, initializing [ 2070.185652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2070.188969] Lustre: 151109:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2070.214072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.987765] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2075.126623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2075.130053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2075.162284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2075.382165] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2077.270762] Lustre: Failing over lustre-OST0000 [ 2077.334974] Lustre: server umount lustre-OST0000 complete [ 2089.978077] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2090.083015] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2090.086537] Lustre: Skipped 1 previous similar message [ 2090.092648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.098837] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2090.504462] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2091.871597] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2093.075757] Lustre: Failing over lustre-MDT0000 [ 2093.168459] Lustre: server umount lustre-MDT0000 complete [ 2095.587098] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2095.590560] Lustre: Skipped 2 previous similar messages [ 2096.159116] Lustre: 147370:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782363013/real 1782363013] req@0000000047ef87f2 x1868943038898688/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1782363029 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2100.704618] LustreError: 150511:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2100.709541] LustreError: 150511:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 2101.727249] Lustre: 147372:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782363018/real 1782363018] req@000000003b9b6424 x1868943038900480/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1782363034 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2105.825139] LustreError: 149152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2105.833198] LustreError: 149152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2105.959944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.001502] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.006886] Lustre: 149145:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.201.141@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2106.087451] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2106.105136] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2106.134509] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2106.286847] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2107.458495] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2111.463463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2111.471357] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2111.474431] Lustre: Skipped 1 previous similar message [ 2111.490613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2116.581070] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2116.581939] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.583250] LustreError: Skipped 1 previous similar message [ 2116.584040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.587741] Lustre: Skipped 1 previous similar message [ 2118.547231] Lustre: server umount lustre-OST0000 complete [ 2121.696768] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2121.697470] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2121.697983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2121.697988] Lustre: Skipped 1 previous similar message [ 2121.709620] Lustre: Skipped 1 previous similar message [ 2126.816171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2126.817898] Lustre: Skipped 2 previous similar messages [ 2131.936295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.939031] Lustre: Skipped 1 previous similar message [ 2133.983106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2134.045961] Lustre: server umount lustre-MDT0000 complete [ 2135.311288] LustreError: 149135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363068 with bad export cookie 2844691424295482839 [ 2135.315082] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2135.315304] LustreError: 149135:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2135.431812] Lustre: server umount lustre-MDT0001 complete [ 2138.861991] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2139.834403] Key type lgssc unregistered [ 2139.967317] LNet: 154756:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2139.970542] LNetError: 154756:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2139.982295] LNet: Removed LNI 192.168.201.141@tcp [ 2140.267591] Key type .llcrypt unregistered [ 2140.268697] Key type ._llcrypt unregistered [ 2160.123502] Key type ._llcrypt registered [ 2160.124635] Key type .llcrypt registered [ 2160.170181] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2160.709861] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2160.741057] alg: No test for adler32 (adler32-zlib) [ 2161.612816] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 2161.713290] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2163.303191] Key type lgssc registered [ 2163.686165] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2166.332527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2166.338306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2167.488163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2168.835751] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2172.652658] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2174.635208] LustreError: 156301:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2174.638538] LustreError: 156301:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2174.641288] LustreError: 156301:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2174.752377] Lustre: server umount lustre-MDT0000 complete [ 2179.827362] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2183.133668] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2194.299597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2194.388410] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2194.397923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2194.432814] Lustre: lustre-MDT0000: new disk, initializing [ 2194.454462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2194.459906] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2195.793364] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2199.793712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2199.831035] Lustre: 159596:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2199.852140] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2199.853815] Lustre: Skipped 1 previous similar message [ 2199.888876] Lustre: lustre-MDT0001: new disk, initializing [ 2199.921309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.934173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2199.937258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2201.253372] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2204.744045] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2206.382289] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2208.570640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2208.679927] Lustre: lustre-OST0000: new disk, initializing [ 2208.682472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2208.686263] Lustre: 161565:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2208.714507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2209.467713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2209.470987] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2209.508792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2210.582035] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2214.123773] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2215.909940] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2216.446567] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 00:52:29 (1782363149) [ 2279.391678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.391678] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2279.391835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2279.398501] Lustre: Skipped 1 previous similar message [ 2279.404164] Lustre: Skipped 1 previous similar message [ 2284.319242] Lustre: server umount lustre-MDT0000 complete [ 2284.511342] LustreError: 159608:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2284.515665] LustreError: 159608:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2285.496537] LustreError: 159589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363218 with bad export cookie 15098083865548714314 [ 2285.497428] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2285.501232] LustreError: 159589:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2289.631462] LustreError: 159605:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2289.631547] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2289.642242] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2289.643932] Lustre: Skipped 1 previous similar message [ 2290.655378] LustreError: 159604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2290.655424] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2291.675509] Lustre: server umount lustre-MDT0001 complete [ 2299.022352] Lustre: server umount lustre-OST0000 complete [ 2301.863991] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2304.914434] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2316.580879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2316.660981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2316.671646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2316.709781] Lustre: lustre-MDT0000: new disk, initializing [ 2316.729146] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2316.734736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2317.944155] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2321.875421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2321.913647] Lustre: 166189:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2321.933341] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2321.935167] Lustre: Skipped 1 previous similar message [ 2321.968949] Lustre: lustre-MDT0001: new disk, initializing [ 2322.011931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2322.014982] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2323.244653] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2326.762537] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2328.444453] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2330.562404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2330.663545] Lustre: lustre-OST0000: new disk, initializing [ 2330.666047] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2330.668800] Lustre: 168156:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2330.695668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2330.697649] Lustre: Skipped 1 previous similar message [ 2332.550569] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2332.981438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2332.984654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2333.016471] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2336.159291] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2337.542258] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2337.616357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2338.936076] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2339.002430] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2343.392688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2343.393758] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2343.395103] LustreError: Skipped 1 previous similar message [ 2343.395706] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2343.399825] Lustre: Skipped 1 previous similar message [ 2345.736984] Lustre: server umount lustre-OST0000 complete [ 2347.487861] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2347.491312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.497340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2347.499085] Lustre: Skipped 1 previous similar message [ 2358.752122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.754227] Lustre: Skipped 5 previous similar messages [ 2361.311106] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2361.371157] Lustre: server umount lustre-MDT0000 complete [ 2362.547183] LustreError: 166182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363295 with bad export cookie 15098083865548873130 [ 2362.550566] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2362.552182] LustreError: 166182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2362.663337] Lustre: server umount lustre-MDT0001 complete [ 2365.302418] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:54:58 (1782363298) [ 2370.117818] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2373.232124] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2383.947353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2384.027304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2384.038521] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2384.074271] Lustre: lustre-MDT0000: new disk, initializing [ 2384.094093] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.099320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2385.461544] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2389.604641] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2389.650749] Lustre: 172901:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2389.765267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2391.129798] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2394.709614] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2396.511493] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2398.726841] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2398.827139] Lustre: 174879:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2400.656386] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2404.163189] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2407.926832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2407.959956] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2413.025964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2413.026047] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2413.029609] LustreError: Skipped 1 previous similar message [ 2413.033013] Lustre: Skipped 1 previous similar message [ 2413.033663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2413.038341] Lustre: Skipped 1 previous similar message [ 2417.101311] Lustre: server umount lustre-OST0000 complete [ 2420.703850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2423.266327] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2423.269695] Lustre: Skipped 2 previous similar messages [ 2424.418940] Lustre: server umount lustre-MDT0000 complete [ 2425.666253] LustreError: 173702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363358 with bad export cookie 15098083865548873998 [ 2425.669945] LustreError: 173702:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2425.670729] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2425.805694] Lustre: server umount lustre-MDT0001 complete [ 2428.491613] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 00:56:02 (1782363362) [ 2433.475837] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2436.368724] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2447.053670] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.134497] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2447.143247] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2447.144849] Lustre: Skipped 3 previous similar messages [ 2447.182179] Lustre: lustre-MDT0000: new disk, initializing [ 2447.183409] Lustre: Skipped 2 previous similar messages [ 2447.202884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2447.204669] Lustre: Skipped 2 previous similar messages [ 2447.209191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2447.212609] Lustre: Skipped 2 previous similar messages [ 2448.501587] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2452.399536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2452.437902] Lustre: 179196:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2452.531310] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2453.735037] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2457.177072] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2458.832801] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2460.860907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2460.957904] Lustre: 181169:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2461.366869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2462.834112] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2467.246935] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2471.911656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2471.913296] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2471.914722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2471.914736] Lustre: Skipped 3 previous similar messages [ 2471.919412] Lustre: Skipped 1 previous similar message [ 2471.924106] LustreError: Skipped 1 previous similar message [ 2475.724799] Lustre: server umount lustre-OST0000 complete [ 2491.359265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2491.469370] Lustre: server umount lustre-MDT0000 complete [ 2493.169683] LustreError: 179186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363426 with bad export cookie 15098083865548875069 [ 2493.173678] LustreError: 179186:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2493.174796] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2493.315457] Lustre: server umount lustre-MDT0001 complete [ 2496.869471] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2498.027327] Key type lgssc unregistered [ 2498.161385] LNet: 183023:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2498.163598] LNetError: 183023:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2498.169436] LNet: Removed LNI 192.168.201.141@tcp [ 2498.508736] Key type .llcrypt unregistered [ 2498.509912] Key type ._llcrypt unregistered [ 2506.705067] Key type ._llcrypt registered [ 2506.706233] Key type .llcrypt registered [ 2506.741946] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2507.053882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2507.077447] alg: No test for adler32 (adler32-zlib) [ 2507.928699] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 2508.020079] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2509.607115] Key type lgssc registered [ 2509.961723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2512.441208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2512.447233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2513.595369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2514.830222] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2517.545232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.670655] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2518.858320] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2520.956178] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2522.533442] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2524.519502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.628159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2526.369381] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2528.539078] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2540.010734] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2558.434345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2558.434522] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2558.436750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2558.442291] Lustre: Skipped 1 previous similar message [ 2561.678802] Lustre: server umount lustre-OST0000 complete [ 2563.552668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2563.556789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2563.558449] Lustre: Skipped 2 previous similar messages [ 2568.672557] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2568.674232] Lustre: Skipped 1 previous similar message [ 2573.792043] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2573.794287] Lustre: Skipped 1 previous similar message [ 2577.375121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2577.442408] Lustre: server umount lustre-MDT0000 complete [ 2578.632278] LustreError: 187220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363511 with bad export cookie 4732988681771974012 [ 2578.635404] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2578.636985] LustreError: 187220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2578.746150] Lustre: server umount lustre-MDT0001 complete [ 2582.175159] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 2583.136359] Key type lgssc unregistered [ 2583.261336] LNet: 187971:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2583.264942] LNetError: 187971:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2583.273315] LNet: Removed LNI 192.168.201.141@tcp [ 2583.582562] Key type .llcrypt unregistered [ 2583.584249] Key type ._llcrypt unregistered [ 2591.845761] Key type ._llcrypt registered [ 2591.846854] Key type .llcrypt registered [ 2591.882275] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2592.322369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2592.335125] alg: No test for adler32 (adler32-zlib) [ 2593.184917] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 2593.266803] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 2594.839125] Key type lgssc registered [ 2595.161026] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2597.413708] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2597.418871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.567946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.868446] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2602.478338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.595051] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2603.803354] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2605.932159] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2607.546745] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2609.623196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2609.731205] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.481614] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2613.636833] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2624.102507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2633.696144] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2633.698470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2633.702080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.723657] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2634.723786] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2634.723824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.723831] Lustre: Skipped 1 previous similar message [ 2637.272317] Lustre: server umount lustre-OST0000 complete [ 2639.564094] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90cd042db000.ost_server_uuid 50 [ 2640.083345] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2653.664375] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2653.670547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2653.675505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2653.677612] Lustre: Skipped 1 previous similar message [ 2656.963462] Lustre: server umount lustre-MDT0000 complete [ 2658.655469] LustreError: 189344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363591 with bad export cookie 10461239832045090981 [ 2658.660616] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2658.662664] LustreError: 189344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2658.834264] Lustre: server umount lustre-MDT0001 complete [ 2662.402619] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 00:59:55 (1782363595) [ 2664.991334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.188217] LustreError: 193276:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2665.195537] LustreError: 193276:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2665.224394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2666.621419] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2669.382754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2669.489144] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2670.812592] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2672.984245] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2674.641700] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2676.828238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.846115] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2681.094651] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2683.189280] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2683.270632] Lustre: lustre-OST0001: new disk, initializing [ 2683.272867] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2683.276354] Lustre: 196115:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2683.307524] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2683.309332] Lustre: Skipped 1 previous similar message [ 2685.144196] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2687.226779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2687.230238] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2687.268272] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2688.695650] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2690.528654] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2690.530931] Lustre: Skipped 2 previous similar messages [ 2690.531196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2690.534688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2690.538286] Lustre: Skipped 1 previous similar message [ 2695.609415] Lustre: server umount lustre-OST0000 complete [ 2695.649577] LustreError: 195096:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2695.655592] LustreError: 195096:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2697.543654] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2697.614109] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2698.931612] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2699.008358] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2699.183801] LustreError: 196922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.304347] LustreError: 196922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.311112] LustreError: 196922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2711.010519] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2711.011660] LustreError: 195094:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.012092] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2711.012100] Lustre: Skipped 2 previous similar messages [ 2711.016566] Lustre: Skipped 1 previous similar message [ 2711.023708] LustreError: 195094:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2715.153330] Lustre: server umount lustre-OST0001 complete [ 2720.735835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2720.738484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2720.742225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2720.744454] Lustre: Skipped 1 previous similar message [ 2722.455588] Lustre: server umount lustre-MDT0000 complete [ 2723.588160] LustreError: 196113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363656 with bad export cookie 10461239832045092038 [ 2723.591499] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.593288] LustreError: 196113:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2723.702911] Lustre: server umount lustre-MDT0001 complete [ 2730.829067] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 01:01:04 (1782363664) [ 2732.805997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2732.891387] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2732.902895] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2732.966098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2734.215944] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2736.918519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2736.949271] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2736.960461] Lustre: 199541:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2738.288721] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2740.618709] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2742.290560] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2744.458328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2744.530954] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2746.429114] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2748.805315] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2750.486448] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2751.721275] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2751.778673] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2754.758294] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2758.682225] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2760.200177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2760.205735] Lustre: Skipped 1 previous similar message [ 2760.208452] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2760.214046] Lustre: Skipped 3 previous similar messages [ 2760.253456] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2766.218762] Lustre: server umount lustre-OST0000 complete [ 2769.327534] LustreError: 201388:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2780.641067] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2783.445803] Lustre: server umount lustre-OST0001 complete [ 2790.807114] Lustre: server umount lustre-MDT0000 complete [ 2790.879825] LustreError: 200294:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2790.884610] LustreError: 200294:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2791.908539] LustreError: 200287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363724 with bad export cookie 10461239832045093200 [ 2791.911156] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2791.912140] LustreError: 200287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2792.018640] Lustre: server umount lustre-MDT0001 complete [ 2799.344488] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 01:02:12 (1782363732) [ 2804.828637] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 2808.111976] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 2819.672726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2819.753246] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2819.764573] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2819.799563] Lustre: lustre-MDT0000: new disk, initializing [ 2819.820413] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2819.822123] Lustre: Skipped 3 previous similar messages [ 2819.827162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2821.174584] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2825.287042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2825.338155] Lustre: 207593:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2825.366704] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2825.370158] Lustre: Skipped 1 previous similar message [ 2825.415905] Lustre: lustre-MDT0001: new disk, initializing [ 2825.456528] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2825.459577] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2826.777686] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2830.460590] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2832.356303] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2834.623306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2834.727020] Lustre: lustre-OST0000: new disk, initializing [ 2834.728680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2834.731139] Lustre: 209560:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2836.686517] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2840.332446] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2841.087283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2841.090426] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2841.136192] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2841.784355] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2841.867810] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2843.181135] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2843.247706] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2846.176376] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2846.179551] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2846.180253] LustreError: Skipped 2 previous similar messages [ 2846.184311] Lustre: Skipped 5 previous similar messages [ 2846.185293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2846.189576] Lustre: Skipped 8 previous similar messages [ 2850.008752] Lustre: server umount lustre-OST0000 complete [ 2851.305443] LustreError: 188676:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@0000000044ab1b7b x1868943616788480/t0(0) o8->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2857.432710] Lustre: server umount lustre-MDT0000 complete [ 2858.701430] LustreError: 207587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363791 with bad export cookie 10461239832045094936 [ 2858.704086] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2858.705675] LustreError: 207587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2858.838404] Lustre: server umount lustre-MDT0001 complete [ 2861.674726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2861.826265] LustreError: 211775:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2863.216795] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2866.108436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2867.586873] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2869.939948] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2871.730484] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2874.022169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2876.094901] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2878.537420] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2879.953545] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2888.178522] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2889.607498] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2889.685709] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2891.744366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2891.748668] LustreError: Skipped 1 previous similar message [ 2896.396094] Lustre: server umount lustre-OST0000 complete [ 2898.309882] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2899.635751] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2923.249726] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2923.364604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2923.367585] Lustre: Skipped 5 previous similar messages [ 2925.389424] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2927.775982] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2928.324087] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2931.341308] Lustre: server umount lustre-OST0000 complete [ 2932.704071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2932.707682] Lustre: Skipped 5 previous similar messages [ 2932.709348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2932.710987] Lustre: Skipped 7 previous similar messages [ 2934.740735] Lustre: server umount lustre-MDT0000 complete [ 2935.925753] LustreError: 216720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363868 with bad export cookie 10461239832045095804 [ 2935.928934] LustreError: 216720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2935.929503] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.037400] Lustre: server umount lustre-MDT0001 complete [ 2938.863816] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 01:04:32 (1782363872) [ 2940.988030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2941.139199] LustreError: 217789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2942.415560] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2945.073234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2946.493339] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2948.775122] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2950.508943] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2952.798248] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.811884] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2957.019839] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2958.379265] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2966.602594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2967.967646] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2968.055178] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2970.221206] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2970.309672] Lustre: lustre-OST0001: new disk, initializing [ 2970.311401] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2970.313854] Lustre: 221212:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2972.230618] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 2975.805534] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 2976.254420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2976.258199] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2976.287734] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2977.215566] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2977.303356] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2978.627112] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2978.698171] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2981.344914] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2981.348589] LustreError: Skipped 2 previous similar messages [ 2985.408740] Lustre: server umount lustre-OST0001 complete [ 2987.485967] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2987.563533] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2988.917760] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2988.995554] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3012.469750] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3014.539087] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3016.889797] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3017.409456] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3017.706583] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3017.710425] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3017.710624] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3024.120189] Lustre: server umount lustre-OST0001 complete [ 3042.123866] Lustre: server umount lustre-OST0000 complete [ 3049.560040] Lustre: server umount lustre-MDT0000 complete [ 3050.722159] LustreError: 217771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782363983 with bad export cookie 10461239832045096735 [ 3050.724829] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3050.725335] LustreError: 217771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3065.311159] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3065.395930] Lustre: server umount lustre-MDT0001 complete [ 3072.703360] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 01:06:46 (1782364006) [ 3074.934761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3075.035615] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3075.048474] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3075.127308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3075.129734] Lustre: Skipped 5 previous similar messages [ 3076.483871] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3079.219589] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3079.249655] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3079.258809] Lustre: 226444:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3080.706118] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3082.978766] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3084.714730] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3086.892636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3086.966249] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3088.862107] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3091.092398] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3094.747508] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3096.668382] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3099.022362] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3100.535112] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 3101.786737] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3101.850238] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3103.117698] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3103.192924] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3104.971804] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90cd04977000.ost_server_uuid 50 [ 3105.479996] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90cd04977000.ost_server_uuid in FULL state after 0 sec [ 3105.820750] Lustre: Permanently deactivating lustre-OST0001 [ 3105.825669] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3106.610163] Lustre: Permanently reactivating lustre-OST0001 [ 3110.369256] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3110.369521] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3110.372337] LustreError: Skipped 7 previous similar messages [ 3110.377911] Lustre: Skipped 8 previous similar messages [ 3110.379984] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3110.382832] Lustre: Skipped 11 previous similar messages [ 3113.720315] Lustre: server umount lustre-OST0001 complete [ 3115.492392] LustreError: 228287:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3115.498770] LustreError: 228287:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 24 previous similar messages [ 3129.311110] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3129.377207] Lustre: server umount lustre-OST0000 complete [ 3145.183178] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3145.260497] Lustre: server umount lustre-MDT0000 complete [ 3146.669104] LustreError: 226436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364079 with bad export cookie 10461239832045098009 [ 3146.674979] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3146.678265] LustreError: 226436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3146.816863] Lustre: server umount lustre-MDT0001 complete [ 3156.077153] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 01:08:09 (1782364089) [ 3159.494069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3159.630979] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3159.644230] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3159.648013] Lustre: Skipped 5 previous similar messages [ 3161.442266] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3165.169061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.206608] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3165.208613] Lustre: Skipped 1 previous similar message [ 3165.220476] Lustre: 233125:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3166.675759] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3169.603672] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3171.709734] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3174.904783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3175.038871] Lustre: Permanently deactivating lustre-OST0000 [ 3175.048361] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3177.258203] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3179.632506] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3182.086576] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3182.092619] LustreError: 235905:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 3182.805365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3182.850347] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3182.855528] Lustre: Skipped 1 previous similar message [ 3185.592567] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3188.308164] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3188.792318] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3189.655812] Lustre: Permanently reactivating lustre-OST0000 [ 3194.379152] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 3194.384123] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3194.391389] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3194.395645] Lustre: Skipped 1 previous similar message [ 3194.422056] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3194.435752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3209.088152] Lustre: server umount lustre-OST0001 complete [ 3216.352162] Lustre: server umount lustre-OST0000 complete [ 3217.966149] Lustre: server umount lustre-MDT0000 complete [ 3219.580404] LustreError: 233117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364152 with bad export cookie 10461239832045099633 [ 3219.584459] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3219.585638] LustreError: 233117:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3225.815739] Lustre: server umount lustre-MDT0001 complete [ 3229.928653] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3230.943977] Key type lgssc unregistered [ 3231.114704] LNet: 237999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3231.120083] LNetError: 237999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3231.131685] LNet: Removed LNI 192.168.201.141@tcp [ 3231.593150] Key type .llcrypt unregistered [ 3231.594881] Key type ._llcrypt unregistered [ 3241.089392] Key type ._llcrypt registered [ 3241.090572] Key type .llcrypt registered [ 3241.140291] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3241.650155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3241.666366] alg: No test for adler32 (adler32-zlib) [ 3242.555770] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3242.658257] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3244.239258] Key type lgssc registered [ 3244.825629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3248.248393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3248.255975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3249.467575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3250.959679] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3254.728969] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3254.767054] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3254.778148] Lustre: 239429:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3254.781635] Lustre: Permanently deactivating lustre-MDT0001 [ 3254.786352] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3254.899765] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3255.247103] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3255.249042] LustreError: 239475:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3255.253069] LustreError: 239475:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3255.255981] LustreError: 239475:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 6, retries 1, failed: rc = -108 [ 3256.536781] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3259.680083] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3261.420170] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3263.940339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.060147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3265.127425] LustreError: 241282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3265.131316] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3265.137236] LustreError: 241282:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3266.222223] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3268.793982] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3270.629139] LustreError: 241282:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3271.623670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3271.717513] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3273.765470] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3274.799228] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3274.807880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3274.829028] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3276.549808] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3278.139322] Lustre: Permanently reactivating lustre-MDT0001 [ 3278.147318] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3278.150444] Lustre: Skipped 2 previous similar messages [ 3282.873318] Lustre: lustre-MDT0001: Client c35b420f-0d94-409c-9fa6-7851154f95a6 (at 192.168.201.41@tcp) reconnecting [ 3285.586587] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3286.206979] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3288.285360] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3288.391353] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3289.854639] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3289.945523] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3290.815613] Lustre: Permanently deactivating lustre-MDT0001 [ 3290.820755] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3290.822913] Lustre: Skipped 2 previous similar messages [ 3300.386820] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3305.442108] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3305.443095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3305.447675] Lustre: Skipped 2 previous similar messages [ 3305.453054] Lustre: Skipped 3 previous similar messages [ 3310.561981] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3311.601928] Lustre: server umount lustre-MDT0000 complete [ 3313.098191] LustreError: 242986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364246 with bad export cookie 499502592736555314 [ 3313.101423] LustreError: 242986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3313.101645] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3313.216779] Lustre: server umount lustre-MDT0001 complete [ 3320.764868] Lustre: server umount lustre-OST0000 complete [ 3325.940562] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 01:10:59 (1782364259) [ 3330.409244] Lustre: server umount lustre-OST0001 complete [ 3332.987671] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3336.835362] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3351.181393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3351.292646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3351.296368] Lustre: Skipped 2 previous similar messages [ 3351.304589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3351.350422] Lustre: lustre-MDT0000: new disk, initializing [ 3351.380156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3351.387211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3353.182155] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3358.418989] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3358.466676] Lustre: 247734:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3358.491615] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3358.493731] Lustre: Skipped 1 previous similar message [ 3358.532176] Lustre: lustre-MDT0001: new disk, initializing [ 3358.587259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3358.591915] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3360.072108] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3364.299661] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3366.329475] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3369.197254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3369.332091] Lustre: lustre-OST0000: new disk, initializing [ 3369.335070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3369.339073] Lustre: 249705:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3369.379753] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3369.382543] Lustre: Skipped 1 previous similar message [ 3371.862025] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3376.212916] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3377.153441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3377.160333] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3377.198590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3382.540700] LustreError: 247741:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3387.206792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3387.282551] Lustre: lustre-OST0001: new disk, initializing [ 3387.284700] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3387.287487] Lustre: 250818:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3387.322831] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3389.397871] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3392.631085] LustreError: 247741:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3392.642641] LustreError: 248835:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3394.146746] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3394.797345] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3395.579887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3395.582907] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3395.610401] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3402.735087] LustreError: 248835:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3405.799669] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3405.800097] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3405.802064] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3405.802077] Lustre: Skipped 3 previous similar messages [ 3405.806605] Lustre: Skipped 2 previous similar messages [ 3405.810773] LustreError: Skipped 1 previous similar message [ 3407.808312] Lustre: lustre-OST0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 3407.813237] Lustre: Skipped 1 previous similar message [ 3409.599708] Lustre: server umount lustre-OST0001 complete [ 3410.920782] LustreError: 249690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3410.931234] LustreError: 249690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3416.042117] LustreError: 249691:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3416.053557] LustreError: 249691:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3421.160886] LustreError: 249690:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3421.167738] LustreError: 249690:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3423.201157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3423.201695] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3423.204859] LustreError: Skipped 1 previous similar message [ 3423.216644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3423.221197] Lustre: Skipped 1 previous similar message [ 3428.128476] Lustre: server umount lustre-OST0000 complete [ 3430.368249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3430.371683] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3430.378108] Lustre: Skipped 1 previous similar message [ 3431.393576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3431.395176] Lustre: Skipped 4 previous similar messages [ 3436.276123] Lustre: server umount lustre-MDT0000 complete [ 3436.512161] LustreError: 247747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3436.517238] LustreError: 247747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 3437.893116] LustreError: 250817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364370 with bad export cookie 499502592736558093 [ 3437.900581] LustreError: 250817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3437.902901] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3452.383248] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3452.508124] Lustre: server umount lustre-MDT0001 complete [ 3456.209321] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3457.105562] Key type lgssc unregistered [ 3457.222385] LNet: 252850:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3457.225435] LNetError: 252850:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3457.237291] LNet: Removed LNI 192.168.201.141@tcp [ 3457.531038] Key type .llcrypt unregistered [ 3457.532259] Key type ._llcrypt unregistered [ 3471.994233] Key type ._llcrypt registered [ 3471.995331] Key type .llcrypt registered [ 3472.045291] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3472.574311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3472.608315] alg: No test for adler32 (adler32-zlib) [ 3473.475366] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3473.576357] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3475.175226] Key type lgssc registered [ 3475.690822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3479.031887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3479.040277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.173518] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3480.188654] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3480.285188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3482.162571] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3485.972602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3486.019231] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3486.030656] Lustre: 255033:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3486.168739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3487.797590] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3490.803678] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3493.316342] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3496.528232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3496.607901] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3496.685431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3499.244624] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3501.500956] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3506.746344] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3547.622858] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3547.625298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3547.627373] LustreError: Skipped 1 previous similar message [ 3547.633641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3547.638689] Lustre: Skipped 1 previous similar message [ 3552.788223] Lustre: server umount lustre-OST0000 complete [ 3554.785091] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3560.905178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3561.065226] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3563.177996] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3566.205799] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3577.450227] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3577.459485] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3577.459569] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3603.426220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.430711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.433209] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3603.439265] Lustre: Skipped 2 previous similar messages [ 3605.687116] Lustre: server umount lustre-OST0000 complete [ 3608.546700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.548975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3608.555490] Lustre: Skipped 1 previous similar message [ 3608.559387] Lustre: Skipped 1 previous similar message [ 3613.623591] Lustre: server umount lustre-MDT0000 complete [ 3613.664499] LustreError: 255043:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3613.670054] LustreError: 255043:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3615.380543] LustreError: 255775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364548 with bad export cookie 1067701697372090738 [ 3615.387982] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3615.389165] LustreError: 255775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3615.589683] Lustre: server umount lustre-MDT0001 complete [ 3620.467756] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3621.796995] Key type lgssc unregistered [ 3621.939504] LNet: 260451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3621.941550] LNetError: 260451:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3621.950415] LNet: Removed LNI 192.168.201.141@tcp [ 3622.248036] Key type .llcrypt unregistered [ 3622.249141] Key type ._llcrypt unregistered [ 3632.684823] Key type ._llcrypt registered [ 3632.686636] Key type .llcrypt registered [ 3632.752298] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3633.153431] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3633.165907] alg: No test for adler32 (adler32-zlib) [ 3634.033113] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3634.121265] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3635.711203] Key type lgssc registered [ 3636.400448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3640.545558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3640.557100] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3641.739489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3643.318351] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3647.221082] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3647.397552] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3648.932666] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3651.992542] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3654.496390] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3657.401926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3657.562193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3659.979001] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3663.055332] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3666.797984] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3677.159936] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3677.167721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3677.168394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.176074] Lustre: Skipped 1 previous similar message [ 3682.274156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3682.275881] Lustre: Skipped 1 previous similar message [ 3682.444561] Lustre: server umount lustre-OST0000 complete [ 3687.394922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3687.402481] Lustre: Skipped 1 previous similar message [ 3687.406086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3687.409674] Lustre: Skipped 1 previous similar message [ 3692.512283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3692.514764] Lustre: Skipped 1 previous similar message [ 3697.633080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3697.636792] Lustre: Skipped 1 previous similar message [ 3698.143126] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3698.231107] Lustre: server umount lustre-MDT0000 complete [ 3699.778258] LustreError: 261824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364632 with bad export cookie 1733319476429310859 [ 3699.780609] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3699.781223] LustreError: 261824:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3699.896095] Lustre: server umount lustre-MDT0001 complete [ 3703.585581] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3705.090925] Key type lgssc unregistered [ 3705.279577] LNet: 266147:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3705.284657] LNetError: 266147:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3705.293839] LNet: Removed LNI 192.168.201.141@tcp [ 3705.782296] Key type .llcrypt unregistered [ 3705.784379] Key type ._llcrypt unregistered [ 3714.056954] Key type ._llcrypt registered [ 3714.058067] Key type .llcrypt registered [ 3714.092540] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3714.403423] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3714.429052] alg: No test for adler32 (adler32-zlib) [ 3715.289515] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3715.389074] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3716.983217] Key type lgssc registered [ 3717.442569] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3720.802726] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3720.814660] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.012911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3723.855817] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3727.714342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3727.923596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3729.935222] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3733.218928] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3735.728699] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3738.799852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3739.021521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3741.820354] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3744.937114] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3746.221068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3761.631939] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3761.633441] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3761.635268] LustreError: Skipped 1 previous similar message [ 3761.642150] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3762.662140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3762.662786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3762.666875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3765.612983] Lustre: server umount lustre-OST0000 complete [ 3767.777251] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3767.778121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3767.781967] Lustre: Skipped 1 previous similar message [ 3767.784222] Lustre: Skipped 1 previous similar message [ 3772.897672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3772.902037] Lustre: Skipped 2 previous similar messages [ 3773.425716] Lustre: server umount lustre-MDT0000 complete [ 3775.252307] LustreError: 267327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364708 with bad export cookie 15391077404951987614 [ 3775.258587] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3775.262216] LustreError: 267327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3775.419680] Lustre: server umount lustre-MDT0001 complete [ 3780.123246] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3781.545336] Key type lgssc unregistered [ 3781.691558] LNet: 271039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3781.694678] LNetError: 271039:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3781.709418] LNet: Removed LNI 192.168.201.141@tcp [ 3782.191419] Key type .llcrypt unregistered [ 3782.192774] Key type ._llcrypt unregistered [ 3792.282909] Key type ._llcrypt registered [ 3792.283899] Key type .llcrypt registered [ 3792.317592] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3792.618422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3792.639726] alg: No test for adler32 (adler32-zlib) [ 3793.511446] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3793.619281] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3795.215233] Key type lgssc registered [ 3795.809052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3799.584477] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3799.595626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3800.806225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3802.715136] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3805.969630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3806.164218] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3807.821211] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3810.833694] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3813.199554] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3816.388068] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3816.572520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3819.149944] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3822.440798] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3825.839064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3842.531705] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3842.533636] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.535533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3842.549343] Lustre: Skipped 1 previous similar message [ 3847.649095] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3847.651625] Lustre: Skipped 2 previous similar messages [ 3848.169576] Lustre: server umount lustre-OST0000 complete [ 3852.256431] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3852.260505] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3852.268056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3856.104906] Lustre: server umount lustre-MDT0000 complete [ 3857.878172] LustreError: 272414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364790 with bad export cookie 4697858188251519251 [ 3857.882953] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3857.885935] LustreError: 272414:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3858.034831] Lustre: server umount lustre-MDT0001 complete [ 3862.597026] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3863.727058] Key type lgssc unregistered [ 3863.856450] LNet: 276758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3863.859117] LNetError: 276758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3863.867395] LNet: Removed LNI 192.168.201.141@tcp [ 3864.210789] Key type .llcrypt unregistered [ 3864.211913] Key type ._llcrypt unregistered [ 3872.720131] Key type ._llcrypt registered [ 3872.721685] Key type .llcrypt registered [ 3872.779091] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3873.156870] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3873.170678] alg: No test for adler32 (adler32-zlib) [ 3874.066880] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3874.189111] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3875.791282] Key type lgssc registered [ 3876.437384] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3880.180529] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3880.191958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3881.462877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3883.370957] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3887.254684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3887.414699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3889.142777] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3892.109312] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3894.462212] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3897.376759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3897.515555] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.257416] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3903.621287] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3906.734580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3914.208381] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3914.213285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3914.221775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3917.283481] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3917.284546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3917.290461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3919.148850] Lustre: server umount lustre-OST0000 complete [ 3922.402321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3922.404159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3922.409952] Lustre: Skipped 1 previous similar message [ 3922.416240] Lustre: Skipped 2 previous similar messages [ 3927.520313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3927.522902] Lustre: Skipped 1 previous similar message [ 3932.641154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3932.645620] Lustre: Skipped 1 previous similar message [ 3935.199130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3935.290589] Lustre: server umount lustre-MDT0000 complete [ 3937.082765] LustreError: 277938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364870 with bad export cookie 8978360108734643358 [ 3937.088705] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3937.090259] LustreError: 277938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3937.246902] Lustre: server umount lustre-MDT0001 complete [ 3941.750390] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 3943.183005] Key type lgssc unregistered [ 3943.341478] LNet: 281672:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3943.344476] LNetError: 281672:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3943.354363] LNet: Removed LNI 192.168.201.141@tcp [ 3943.726183] Key type .llcrypt unregistered [ 3943.728995] Key type ._llcrypt unregistered [ 3948.832561] Key type ._llcrypt registered [ 3948.834384] Key type .llcrypt registered [ 3948.894504] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3956.967395] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 3964.438598] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 3964.713761] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3964.727618] alg: No test for adler32 (adler32-zlib) [ 3965.580444] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 3965.669060] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 3967.263221] Key type lgssc registered [ 3967.813337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3982.706353] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3982.717122] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3983.846248] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3983.863171] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3983.917391] Lustre: lustre-MDT0000: new disk, initializing [ 3983.949327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3983.956472] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3985.374710] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3990.413384] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3990.494499] Lustre: 285196:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3990.542579] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3990.546741] Lustre: Skipped 1 previous similar message [ 3990.598705] Lustre: lustre-MDT0001: new disk, initializing [ 3990.638231] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3990.653655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3990.657455] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3992.226697] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 3996.297359] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3998.341421] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4001.082642] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4001.220962] Lustre: lustre-OST0000: new disk, initializing [ 4001.223117] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4001.226112] Lustre: 287166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4001.261570] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4003.142106] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4005.390413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4005.397403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4005.451943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4007.967950] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4009.263275] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4009.338701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4011.060492] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4011.166744] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4015.589438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4015.589893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.596858] LustreError: Skipped 1 previous similar message [ 4015.611281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4015.616421] Lustre: Skipped 1 previous similar message [ 4020.709194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4020.713149] Lustre: Skipped 1 previous similar message [ 4025.829515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4025.833891] Lustre: Skipped 1 previous similar message [ 4026.335130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4026.420681] Lustre: server umount lustre-OST0000 complete [ 4030.946127] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.947819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4030.953979] Lustre: Skipped 2 previous similar messages [ 4034.357689] Lustre: server umount lustre-MDT0000 complete [ 4036.107187] LustreError: 285991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782364969 with bad export cookie 8288980998150000499 [ 4036.110180] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4036.112473] LustreError: 285991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4036.222336] Lustre: server umount lustre-MDT0001 complete [ 4040.009747] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 01:22:53 (1782364973) [ 4043.186034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4043.424983] LustreError: 289575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4043.434580] LustreError: 289575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4043.459481] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4044.978642] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4048.152565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4048.293722] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4049.739425] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4052.951430] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4055.664793] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4058.915265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4059.081755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4061.721490] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4065.116865] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4078.563057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4078.564200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4078.571883] LustreError: Skipped 1 previous similar message [ 4078.573070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4078.576881] Lustre: Skipped 2 previous similar messages [ 4084.649469] Lustre: server umount lustre-OST0000 complete [ 4088.801848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.808727] Lustre: Skipped 1 previous similar message [ 4088.811569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4088.814665] Lustre: Skipped 3 previous similar messages [ 4092.721826] Lustre: server umount lustre-MDT0000 complete [ 4093.920917] LustreError: 289576:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4093.930432] LustreError: 289576:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4094.519999] LustreError: 289557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365027 with bad export cookie 8288980998150001388 [ 4094.526589] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4094.528112] LustreError: 289557:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4094.661723] Lustre: server umount lustre-MDT0001 complete [ 4099.252284] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4100.276446] Key type lgssc unregistered [ 4100.423591] LNet: 293131:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4100.426684] LNetError: 293131:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4100.439620] LNet: Removed LNI 192.168.201.141@tcp [ 4100.844158] Key type .llcrypt unregistered [ 4100.846259] Key type ._llcrypt unregistered [ 4112.021372] Key type ._llcrypt registered [ 4112.022861] Key type .llcrypt registered [ 4112.085734] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 4119.521907] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4120.029727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4120.084927] alg: No test for adler32 (adler32-zlib) [ 4120.969243] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4121.067385] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4122.663260] Key type lgssc registered [ 4123.283954] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4145.035121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4145.045681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4146.153275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4146.171472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4146.217028] Lustre: lustre-MDT0000: new disk, initializing [ 4146.245725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4146.254390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4148.244313] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4153.709940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4153.746075] Lustre: 296929:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4153.766476] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4153.768124] Lustre: Skipped 1 previous similar message [ 4153.805721] Lustre: lustre-MDT0001: new disk, initializing [ 4153.836922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4153.851125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4153.853911] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4155.498610] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4160.333698] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4162.837881] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4165.839635] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4165.965932] Lustre: lustre-OST2710: new disk, initializing [ 4165.970005] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4165.976251] Lustre: 298899:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4166.013963] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4168.547507] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4173.387339] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4174.345504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4174.352102] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4174.409407] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4176.768045] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4176.879121] Lustre: lustre-OST03e8: new disk, initializing [ 4176.882313] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4176.887233] Lustre: 299963:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4176.939774] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4179.657269] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4179.930206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4179.939666] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4179.986666] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4183.150414] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4186.458266] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4186.578251] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4188.469961] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4188.574685] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4190.376485] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4190.468316] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4192.503200] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4192.629543] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4194.785685] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4194.790716] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4194.799498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.300359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.302728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.315444] Lustre: Skipped 1 previous similar message [ 4200.253768] Lustre: server umount lustre-MDT0000 complete [ 4201.995407] LustreError: 296965:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365134 with bad export cookie 4977270737117095399 [ 4202.001409] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4202.139286] Lustre: server umount lustre-MDT0001 complete [ 4213.783124] Lustre: server umount lustre-OST2710 complete [ 4225.907366] Lustre: server umount lustre-OST03e8 complete [ 4234.344844] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 4239.043932] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4256.916045] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4257.013689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4257.023351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4257.064363] Lustre: lustre-MDT0000: new disk, initializing [ 4257.087891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4257.093782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4258.839069] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4264.128876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4264.195017] Lustre: 305212:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4264.327993] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4266.231426] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4271.178386] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4273.967330] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4277.307069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4277.457094] Lustre: lustre-OST0000: new disk, initializing [ 4277.460080] Lustre: Skipped 1 previous similar message [ 4277.463402] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4277.467444] Lustre: Skipped 2 previous similar messages [ 4277.471447] Lustre: 307184:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4277.517739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4277.522049] Lustre: Skipped 1 previous similar message [ 4279.790187] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4284.227620] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4284.409784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4284.412249] Lustre: Skipped 1 previous similar message [ 4284.414665] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4284.447615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4285.706542] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4285.771369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4287.029248] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4287.087040] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4289.506887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4289.507233] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4289.510557] Lustre: Skipped 1 previous similar message [ 4289.512698] LustreError: Skipped 1 previous similar message [ 4289.513261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4289.516587] Lustre: Skipped 4 previous similar messages [ 4293.774854] Lustre: server umount lustre-OST0000 complete [ 4299.747715] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4299.749258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4299.755658] Lustre: Skipped 1 previous similar message [ 4299.761761] Lustre: Skipped 1 previous similar message [ 4301.571507] Lustre: server umount lustre-MDT0000 complete [ 4303.427842] LustreError: 305204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365236 with bad export cookie 4977270737117097065 [ 4303.435076] LustreError: 305204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4303.435928] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4303.614076] Lustre: server umount lustre-MDT0001 complete [ 4307.612218] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 01:27:20 (1782365240) [ 4308.210900] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4308.861984] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 01:27:22 (1782365242) [ 4310.234108] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4322.837326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4322.944905] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4322.952386] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4323.011129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4324.812809] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4329.114033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4329.160286] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4329.172162] Lustre: 311031:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4331.032742] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4348.196440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4348.306235] Lustre: Denying initial registration attempt from nid 192.168.201.141@tcp, specified as failover [ 4348.310689] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4348.316088] LustreError: 312485:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4348.320195] LustreError: 312485:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4348.323617] LustreError: 312485:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4348.353703] Lustre: server umount lustre-OST0000 complete [ 4348.355718] LustreError: 312485:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4355.040415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4355.045056] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.053216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4357.676117] Lustre: server umount lustre-MDT0000 complete [ 4359.471329] LustreError: 311067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365292 with bad export cookie 4977270737117097933 [ 4359.478201] LustreError: 311067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4359.478299] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4359.616722] Lustre: server umount lustre-MDT0001 complete [ 4364.642696] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4366.248711] Key type lgssc unregistered [ 4366.428729] LNet: 313603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4366.433770] LNetError: 313603:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4366.442788] LNet: Removed LNI 192.168.201.141@tcp [ 4366.943986] Key type .llcrypt unregistered [ 4366.945846] Key type ._llcrypt unregistered [ 4371.965247] Key type ._llcrypt registered [ 4371.966784] Key type .llcrypt registered [ 4372.021893] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4372.536797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4372.547428] alg: No test for adler32 (adler32-zlib) [ 4373.384483] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4373.462298] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4375.031180] Key type lgssc registered [ 4375.603924] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4401.184442] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4404.773683] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4404.781816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4405.920236] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4405.931602] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4406.011730] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4407.801639] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4413.911223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4413.976583] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4413.992993] Lustre: 316340:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4414.173123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4417.516542] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4422.547678] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4426.826710] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4444.643540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4444.746825] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4444.835355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4447.413871] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4450.103516] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4450.795449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4450.812123] LustreError: 318190:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4450.818848] LustreError: 318190:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4450.841152] Lustre: server umount lustre-OST0000 complete [ 4455.391760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4455.393801] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4455.397566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4455.904428] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4458.133075] Lustre: server umount lustre-MDT0000 complete [ 4459.269128] LustreError: 316333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365392 with bad export cookie 14989054521721838483 [ 4459.272167] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4459.272676] LustreError: 316333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4459.387349] Lustre: server umount lustre-MDT0001 complete [ 4462.739789] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4463.777334] Key type lgssc unregistered [ 4463.913493] LNet: 319907:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4463.918538] LNetError: 319907:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4463.931718] LNet: Removed LNI 192.168.201.141@tcp [ 4464.256771] Key type .llcrypt unregistered [ 4464.258367] Key type ._llcrypt unregistered [ 4473.149231] Key type ._llcrypt registered [ 4473.150676] Key type .llcrypt registered [ 4473.197716] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4473.537608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4473.605042] alg: No test for adler32 (adler32-zlib) [ 4474.457151] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4474.547484] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4476.135177] Key type lgssc registered [ 4476.538407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4479.104693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4479.110114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4480.244371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4481.554863] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4484.320324] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4484.451637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4485.719621] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4488.052402] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4489.854258] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4493.986024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4494.094444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4495.936148] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4498.233995] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4510.688776] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.689898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4510.691849] Lustre: Skipped 1 previous similar message [ 4513.716363] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 4513.718393] Lustre: Skipped 1 previous similar message [ 4515.295870] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4515.297757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4515.301841] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4516.758508] Lustre: server umount lustre-MDT0000 complete [ 4518.038448] LustreError: 321282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365450 with bad export cookie 3964773737829707924 [ 4518.041337] LustreError: 321282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4518.041985] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4518.830460] LustreError: 321296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4518.836143] LustreError: 321296:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4518.837858] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 4518.842548] Lustre: Skipped 2 previous similar messages [ 4520.927929] LustreError: 321295:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4520.928189] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4523.935860] LustreError: 321296:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4523.936125] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4523.944208] Lustre: Skipped 1 previous similar message [ 4524.204994] Lustre: server umount lustre-MDT0001 complete [ 4526.426279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4529.388321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4542.431598] LustreError: 320612:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a423b937 x1868945589034496/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4542.524550] LustreError: 323129:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4542.529372] LustreError: 323129:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4542.545133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4542.564284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4543.787315] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4546.356966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4547.490221] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4547.694955] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4549.891886] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4552.614208] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4555.488225] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 5 sec [ 4557.060721] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4557.615854] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4561.888040] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4561.890024] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4561.893555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4561.896210] Lustre: Skipped 1 previous similar message [ 4562.915055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4573.151119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4573.213984] Lustre: server umount lustre-OST0000 complete [ 4577.247850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4577.250082] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.253072] Lustre: Skipped 1 previous similar message [ 4578.272079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4578.273932] Lustre: Skipped 8 previous similar messages [ 4580.568145] Lustre: server umount lustre-MDT0000 complete [ 4581.755366] LustreError: 321281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365514 with bad export cookie 3964773737829715078 [ 4581.758263] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4581.759316] LustreError: 321281:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4581.872830] Lustre: server umount lustre-MDT0001 complete [ 4585.521480] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4586.565689] Key type lgssc unregistered [ 4586.697363] LNet: 327582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4586.700166] LNetError: 327582:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4586.709448] LNet: Removed LNI 192.168.201.141@tcp [ 4587.026260] Key type .llcrypt unregistered [ 4587.028033] Key type ._llcrypt unregistered [ 4596.094978] Key type ._llcrypt registered [ 4596.096560] Key type .llcrypt registered [ 4596.139493] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4596.535118] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4596.577368] alg: No test for adler32 (adler32-zlib) [ 4597.432921] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4597.521246] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4599.103098] Key type lgssc registered [ 4599.453357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4602.028647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4602.034196] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4603.197563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.618562] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4607.375288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4607.508749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4608.787483] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4611.321606] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4613.236148] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4617.313027] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.316734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4617.318258] Lustre: Skipped 1 previous similar message [ 4620.383494] Lustre: server umount lustre-MDT0000 complete [ 4621.560489] LustreError: 328955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365554 with bad export cookie 17700914533961073756 [ 4621.562405] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4621.564025] LustreError: 328955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4621.679723] Lustre: server umount lustre-MDT0001 complete [ 4627.069839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4627.184280] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4627.196722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4627.284909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4628.563338] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4631.697033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4631.741237] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4631.756105] Lustre: 331560:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4631.881650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4633.338115] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4636.332076] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4638.358695] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4643.105851] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4643.174475] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4643.177540] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4643.180294] LustreError: 333440:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4643.182442] LustreError: 333440:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4643.184444] LustreError: 333440:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4643.206304] Lustre: server umount lustre-OST0000 complete [ 4643.207670] LustreError: 333440:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4648.162842] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4648.234710] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4648.308450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4650.214448] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4652.591606] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4654.109647] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4655.305103] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4655.348357] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4655.388937] Lustre: lustre-OST0001: new disk, initializing [ 4655.390352] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4655.394384] Lustre: 334976:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4657.453066] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4659.905341] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4660.740312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4660.743647] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4660.786470] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4665.828727] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4665.830853] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.833578] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4665.833959] Lustre: Skipped 2 previous similar messages [ 4667.011268] Lustre: server umount lustre-OST0001 complete [ 4669.408342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4669.412703] LustreError: Skipped 1 previous similar message [ 4669.415261] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4669.422777] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4669.426184] Lustre: Skipped 2 previous similar messages [ 4670.946996] LustreError: 335754:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.950939] LustreError: 335754:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4674.314591] Lustre: server umount lustre-OST0000 complete [ 4676.064743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4676.065366] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4676.068560] Lustre: Skipped 2 previous similar messages [ 4676.070646] Lustre: Skipped 2 previous similar messages [ 4681.184390] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4681.186402] Lustre: Skipped 2 previous similar messages [ 4681.623331] Lustre: server umount lustre-MDT0000 complete [ 4682.859995] LustreError: 333921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365615 with bad export cookie 17700914533961074288 [ 4682.863296] LustreError: 333921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4682.863934] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4682.996578] Lustre: server umount lustre-MDT0001 complete [ 4686.453719] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4687.412103] Key type lgssc unregistered [ 4687.531289] LNet: 336935:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4687.533441] LNetError: 336935:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4687.544278] LNet: Removed LNI 192.168.201.141@tcp [ 4687.829174] Key type .llcrypt unregistered [ 4687.830193] Key type ._llcrypt unregistered [ 4718.020676] Key type ._llcrypt registered [ 4718.022272] Key type .llcrypt registered [ 4718.065516] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 4724.343857] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4724.676588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4724.710896] alg: No test for adler32 (adler32-zlib) [ 4725.568388] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4725.657809] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4727.239400] Key type lgssc registered [ 4727.580344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4739.043882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4739.049371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.134400] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4740.144932] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4740.179641] Lustre: lustre-MDT0000: new disk, initializing [ 4740.199569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4740.205369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4741.500764] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4746.098469] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4746.154660] Lustre: 341834:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4746.191177] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4746.194909] Lustre: Skipped 1 previous similar message [ 4746.240557] Lustre: lustre-MDT0001: new disk, initializing [ 4746.278079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4746.291631] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4746.294385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4747.688054] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4751.443251] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4753.137557] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4755.273241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4755.375443] Lustre: lustre-OST0000: new disk, initializing [ 4755.377697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4755.380571] Lustre: 343799:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4755.407639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4757.221206] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4759.488370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4759.493804] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4759.539409] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4760.855628] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4762.241564] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4762.317715] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4763.694259] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4763.777175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4764.643192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4764.644396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4764.645326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4764.645776] LustreError: Skipped 1 previous similar message [ 4764.655464] Lustre: Skipped 1 previous similar message [ 4766.482488] Lustre: server umount lustre-OST0000 complete [ 4769.760902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4769.761718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4769.764249] Lustre: Skipped 1 previous similar message [ 4769.765854] Lustre: Skipped 1 previous similar message [ 4773.856306] Lustre: server umount lustre-MDT0000 complete [ 4775.133450] LustreError: 342629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365708 with bad export cookie 11011401899544846244 [ 4775.137398] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4775.138968] LustreError: 342629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4775.267888] Lustre: server umount lustre-MDT0001 complete [ 4778.112038] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 01:35:11 (1782365711) [ 4780.516396] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 01:35:14 (1782365714) [ 4783.261015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4783.407930] LustreError: 346551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4783.416060] LustreError: 346551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4783.428766] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4784.801385] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4787.561596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4787.693899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4789.032899] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4791.357545] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4793.127606] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4795.337636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4797.343039] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4799.760844] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4813.635681] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4814.273216] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4814.867298] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4815.452212] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4816.216889] Lustre: Failing over lustre-MDT0000 [ 4816.371776] Lustre: server umount lustre-MDT0000 complete [ 4818.400281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4818.402379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4818.405688] LustreError: 346551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.424800] LustreError: 346570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4819.429458] LustreError: 346570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4824.544124] LustreError: 346552:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4824.549150] LustreError: 346552:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4829.665477] LustreError: 346551:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4829.670149] LustreError: 346551:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4829.717128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4829.755059] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4829.855603] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4829.857428] Lustre: Skipped 1 previous similar message [ 4829.889577] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4830.639287] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4831.458734] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4835.300941] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4835.310864] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4835.328909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4837.355891] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4838.017778] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4840.166479] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4840.976390] Lustre: Failing over lustre-MDT0000 [ 4841.099339] Lustre: server umount lustre-MDT0000 complete [ 4845.537235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4845.538589] LustreError: 346547:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4845.538917] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4845.542607] Lustre: Skipped 4 previous similar messages [ 4845.546559] LustreError: 346547:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4854.498378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4854.545509] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4854.658467] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4854.683404] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4856.017962] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4856.239133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4859.878206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4859.880165] Lustre: Skipped 2 previous similar messages [ 4859.890046] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4859.907922] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4861.637366] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4862.175473] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4864.994230] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4864.994839] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4864.996383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4864.996397] Lustre: Skipped 1 previous similar message [ 4865.006528] Lustre: Skipped 2 previous similar messages [ 4870.112548] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4870.116072] Lustre: Skipped 4 previous similar messages [ 4870.548790] Lustre: server umount lustre-MDT0000 complete [ 4871.741196] LustreError: 346532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365804 with bad export cookie 11011401899544848925 [ 4871.746612] LustreError: 346532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4871.746702] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4871.876675] Lustre: server umount lustre-MDT0001 complete [ 4883.358670] Lustre: server umount lustre-OST0000 complete [ 4888.498697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4888.661415] LustreError: 352903:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4888.666575] LustreError: 352903:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 4888.686663] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4889.931225] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4892.724262] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4894.123037] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4896.396894] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4898.164682] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4900.489953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4902.623336] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4904.974701] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4914.983953] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4918.239868] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4918.241984] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4918.245557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4922.833487] Lustre: server umount lustre-OST0000 complete [ 4930.140413] Lustre: server umount lustre-MDT0000 complete [ 4930.528334] LustreError: 352922:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4930.533402] LustreError: 352922:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4931.341397] LustreError: 352883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365864 with bad export cookie 11011401899544849646 [ 4931.346344] LustreError: 352883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4931.346854] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4931.468186] Lustre: server umount lustre-MDT0001 complete [ 4934.773673] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 4935.822935] Key type lgssc unregistered [ 4935.942486] LNet: 356452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4935.944441] LNetError: 356452:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4935.957503] LNet: Removed LNI 192.168.201.141@tcp [ 4936.293135] Key type .llcrypt unregistered [ 4936.294400] Key type ._llcrypt unregistered [ 4945.574116] Key type ._llcrypt registered [ 4945.575313] Key type .llcrypt registered [ 4945.613592] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 4946.062804] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4946.100670] alg: No test for adler32 (adler32-zlib) [ 4946.955202] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 4947.042736] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 4948.623217] Key type lgssc registered [ 4949.027430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4951.695866] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4951.702717] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.843547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4954.246313] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4957.979773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4958.146428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4959.891398] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4962.777371] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4964.488924] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4966.652634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4966.777816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4968.643202] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 4970.887348] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4978.027550] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4988.387123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4988.389474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4988.391925] Lustre: Skipped 2 previous similar messages [ 4988.394027] Lustre: Skipped 1 previous similar message [ 4991.135979] Lustre: server umount lustre-MDT0000 complete [ 4992.337378] LustreError: 357970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365925 with bad export cookie 9597288846298844523 [ 4992.340739] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4992.340883] LustreError: 357970:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4992.464089] Lustre: server umount lustre-MDT0001 complete [ 5003.938315] Lustre: server umount lustre-OST0000 complete [ 5008.169708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5008.309931] LustreError: 361675:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.314969] LustreError: 361675:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5008.330821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5009.567195] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5012.182049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5013.564882] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5015.905759] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5017.632117] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5020.574528] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5020.757559] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5020.761300] Lustre: Skipped 1 previous similar message [ 5022.919499] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5025.769821] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5034.093147] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5044.705333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5044.706562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5044.708705] Lustre: Skipped 1 previous similar message [ 5044.711928] Lustre: Skipped 2 previous similar messages [ 5049.632386] Lustre: server umount lustre-MDT0000 complete [ 5049.824016] LustreError: 361694:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5049.828179] LustreError: 361694:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5050.973114] LustreError: 361656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782365983 with bad export cookie 9597288846298845622 [ 5050.976171] LustreError: 361656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5050.976321] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5051.085093] Lustre: server umount lustre-MDT0001 complete [ 5062.692774] Lustre: server umount lustre-OST0000 complete [ 5068.875702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5069.053895] LustreError: 365603:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5069.079682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5070.639769] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5074.379332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5074.400966] LustreError: 365604:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5076.357502] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5079.463997] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5081.450201] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5083.785497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5086.010726] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5088.658046] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5093.162732] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5095.392170] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5095.394246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5095.397349] Lustre: Skipped 1 previous similar message [ 5095.399372] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5098.465312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5098.466914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5098.467738] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.398328] Lustre: server umount lustre-OST0000 complete [ 5103.584648] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5103.585398] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5103.587671] Lustre: Skipped 1 previous similar message [ 5103.590257] Lustre: Skipped 2 previous similar messages [ 5108.757889] Lustre: server umount lustre-MDT0000 complete [ 5109.989187] LustreError: 365585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366042 with bad export cookie 9597288846298846630 [ 5109.992603] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5109.995136] LustreError: 365585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5110.109486] Lustre: server umount lustre-MDT0001 complete [ 5113.769283] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5114.764390] Key type lgssc unregistered [ 5114.897388] LNet: 369130:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5114.899571] LNetError: 369130:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5114.912382] LNet: Removed LNI 192.168.201.141@tcp [ 5115.234250] Key type .llcrypt unregistered [ 5115.235216] Key type ._llcrypt unregistered [ 5124.586846] Key type ._llcrypt registered [ 5124.587882] Key type .llcrypt registered [ 5124.633829] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5125.074270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5125.101766] alg: No test for adler32 (adler32-zlib) [ 5125.964338] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5126.059442] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5127.639161] Key type lgssc registered [ 5127.996723] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5130.653706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5130.659495] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.662106] LustreError: 370550:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5130.669868] LustreError: 370550:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5130.672037] LustreError: 370550:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5130.674080] LustreError: 370550:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5130.676318] LustreError: 370550:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5133.178483] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5133.181166] LustreError: 370946:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5133.189092] LustreError: 370946:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5133.192538] LustreError: 370946:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5133.196056] LustreError: 370946:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5133.199972] LustreError: 370946:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5139.718465] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5140.691572] Key type lgssc unregistered [ 5140.810357] LNet: 371946:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5140.812729] LNetError: 371946:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5140.824296] LNet: Removed LNI 192.168.201.141@tcp [ 5141.119686] Key type .llcrypt unregistered [ 5141.120654] Key type ._llcrypt unregistered [ 5146.917317] Key type ._llcrypt registered [ 5146.918985] Key type .llcrypt registered [ 5146.953771] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 5153.295210] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5153.701571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5153.714676] alg: No test for adler32 (adler32-zlib) [ 5154.556639] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5154.640883] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5156.215123] Key type lgssc registered [ 5156.554208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5167.504482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5167.509399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5168.592147] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5168.601523] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5168.643116] Lustre: lustre-MDT0000: new disk, initializing [ 5168.664502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.671609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5169.912520] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5174.471905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5174.518989] Lustre: 375076:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5174.542218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5174.543736] Lustre: Skipped 1 previous similar message [ 5174.577448] Lustre: lustre-MDT0001: new disk, initializing [ 5174.615499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5174.627083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5174.629757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5176.080170] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5179.827414] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5181.768822] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5183.936383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5184.040129] Lustre: lustre-OST0000: new disk, initializing [ 5184.041823] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5184.044548] Lustre: 377046:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5184.073138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5186.039461] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5189.782571] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5191.164348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5191.165318] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5191.167654] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5191.203504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5192.268724] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5193.557671] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5193.626106] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5196.257937] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5196.259124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5196.260115] LustreError: Skipped 1 previous similar message [ 5196.263488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5196.265566] Lustre: Skipped 1 previous similar message [ 5200.404755] Lustre: server umount lustre-OST0000 complete [ 5205.472226] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5205.476788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5205.483849] Lustre: Skipped 1 previous similar message [ 5205.486974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5206.496234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5206.496816] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5211.616392] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5211.618851] Lustre: Skipped 2 previous similar messages [ 5216.223094] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5216.290523] Lustre: server umount lustre-MDT0000 complete [ 5217.502126] LustreError: 377045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366150 with bad export cookie 12427162984460714062 [ 5217.505722] LustreError: 377045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5217.506095] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5217.627600] Lustre: server umount lustre-MDT0001 complete [ 5220.715098] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5221.280821] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 01:42:34 (1782366154) [ 5224.001424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.223875] LustreError: 379512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5224.232731] LustreError: 379512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5224.254529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5225.726229] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5228.489708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5228.615482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5229.875208] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5232.286795] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5233.933209] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5235.943498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5237.732866] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5239.904894] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5242.068240] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5242.125342] Lustre: lustre-OST0001: new disk, initializing [ 5242.126789] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5242.129099] Lustre: 382343:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5242.164896] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5242.168083] Lustre: Skipped 1 previous similar message [ 5243.917522] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5247.568270] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5250.052200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5250.055038] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5250.086991] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5255.139646] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5255.140609] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5255.141827] LustreError: Skipped 1 previous similar message [ 5255.141982] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5255.147174] Lustre: Skipped 1 previous similar message [ 5259.771892] Lustre: server umount lustre-OST0001 complete [ 5260.259837] LustreError: 383152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5260.266716] LustreError: 383152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5265.378397] LustreError: 383152:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5265.382776] LustreError: 383152:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5270.499168] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5270.504911] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5274.079712] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5274.081590] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5274.084907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5274.086268] Lustre: Skipped 2 previous similar messages [ 5275.618224] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5275.622273] LustreError: 381331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5277.766824] Lustre: server umount lustre-OST0000 complete [ 5279.711780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5279.713715] LustreError: Skipped 1 previous similar message [ 5285.013051] Lustre: server umount lustre-MDT0000 complete [ 5285.855930] LustreError: 379513:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5285.860992] LustreError: 379513:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5286.175163] LustreError: 382342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366219 with bad export cookie 12427162984460714944 [ 5286.177386] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.181114] LustreError: 382342:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5292.330655] Lustre: server umount lustre-MDT0001 complete [ 5296.266577] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5297.236189] Key type lgssc unregistered [ 5297.363304] LNet: 384328:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5297.365458] LNetError: 384328:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5297.373274] LNet: Removed LNI 192.168.201.141@tcp [ 5297.666391] Key type .llcrypt unregistered [ 5297.667362] Key type ._llcrypt unregistered [ 5314.678121] Key type ._llcrypt registered [ 5314.679685] Key type .llcrypt registered [ 5314.714967] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5315.146791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5315.168041] alg: No test for adler32 (adler32-zlib) [ 5316.011299] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5316.097538] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5317.671125] Key type lgssc registered [ 5318.024799] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5320.460392] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5320.465846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5321.546256] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5321.552217] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5321.602231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5322.857601] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5325.511013] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5325.540888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5325.553547] Lustre: 387101:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5325.642868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5326.918792] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5329.144658] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5330.871715] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5331.936110] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5331.936677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5331.939312] Lustre: Skipped 1 previous similar message [ 5331.941897] Lustre: Skipped 1 previous similar message [ 5337.056105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5337.058432] Lustre: Skipped 1 previous similar message [ 5337.630211] Lustre: server umount lustre-MDT0000 complete [ 5338.817178] LustreError: 387092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366271 with bad export cookie 17142774233333835626 [ 5338.819866] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5338.820109] LustreError: 387092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5338.921571] Lustre: server umount lustre-MDT0001 complete [ 5341.827641] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 01:44:35 (1782366275) [ 5343.938485] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5344.102575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5345.333013] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5347.940935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.050541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5349.262048] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5351.399027] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5353.048424] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5355.260611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5355.324712] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5355.382619] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5357.565222] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5360.379941] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5364.465117] Lustre: Permanently deactivating lustre-OST0000 [ 5364.470231] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5367.735815] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5369.002211] LustreError: 392544:0:(mgs_llog.c:2041:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5369.009901] LustreError: 392544:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5369.015375] LustreError: 392544:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5369.649582] LustreError: 392592:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5369.651445] LustreError: 392592:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5376.469310] Lustre: server umount lustre-OST0000 complete [ 5377.367873] LustreError: 392842:0:(mgs_llog.c:2096:mgs_replace_nids()) Only MGS is allowed to be started [ 5377.371012] LustreError: 392842:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5378.528109] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.528620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.531209] Lustre: Skipped 1 previous similar message [ 5378.534020] Lustre: Skipped 1 previous similar message [ 5383.648152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5383.651128] Lustre: Skipped 1 previous similar message [ 5388.768099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5388.769767] Lustre: Skipped 1 previous similar message [ 5392.351064] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5392.407757] Lustre: server umount lustre-MDT0000 complete [ 5393.556199] LustreError: 392297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366326 with bad export cookie 17142774233333836193 [ 5393.558591] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5393.560361] LustreError: 392297:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5403.149417] Lustre: server umount lustre-MDT0001 complete [ 5405.740410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.979446] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5407.795668] LustreError: 393998:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5409.727017] Lustre: 394337:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.141@tcp [ 5410.347818] Lustre: server umount MGS complete [ 5413.901413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5414.043099] LustreError: 395058:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.048085] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5414.048270] LustreError: 395058:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5414.050145] Lustre: Skipped 1 previous similar message [ 5414.071052] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5415.294323] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5417.977995] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5418.072038] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5419.356385] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5421.668278] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5423.461068] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5424.081740] Lustre: Permanently reactivating lustre-OST0000 [ 5424.086206] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5424.088334] Lustre: Skipped 2 previous similar messages [ 5450.207092] LustreError: 396887:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5452.808295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5453.004795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5453.010500] Lustre: Skipped 1 previous similar message [ 5455.492044] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5458.608334] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5474.784169] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5474.787467] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5474.792472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5475.813583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5485.023236] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5485.091228] Lustre: server umount lustre-OST0000 complete [ 5489.632636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5489.636904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5489.644129] Lustre: Skipped 1 previous similar message [ 5491.168350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5491.170355] Lustre: Skipped 6 previous similar messages [ 5492.519716] Lustre: server umount lustre-MDT0000 complete [ 5493.998797] LustreError: 395041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366426 with bad export cookie 17142774233333837229 [ 5494.006334] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5494.142885] Lustre: server umount lustre-MDT0001 complete [ 5498.308508] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5499.525343] Key type lgssc unregistered [ 5499.672324] LNet: 399734:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5499.675861] LNetError: 399734:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5499.684289] LNet: Removed LNI 192.168.201.141@tcp [ 5500.100602] Key type .llcrypt unregistered [ 5500.102700] Key type ._llcrypt unregistered [ 5506.393947] Key type ._llcrypt registered [ 5506.394920] Key type .llcrypt registered [ 5506.427394] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 5512.924668] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5513.258571] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5513.306350] alg: No test for adler32 (adler32-zlib) [ 5514.160256] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5514.249932] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5515.831275] Key type lgssc registered [ 5516.277771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5529.063535] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5529.067901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5530.151902] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5530.161069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5530.195313] Lustre: lustre-MDT0000: new disk, initializing [ 5530.215186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5530.220859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5531.608029] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5536.092953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5536.131642] Lustre: 402863:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5536.154156] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5536.155976] Lustre: Skipped 1 previous similar message [ 5536.192724] Lustre: lustre-MDT0001: new disk, initializing [ 5536.223915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5536.258368] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5536.261228] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5537.496747] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5541.316660] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5543.366723] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5546.256896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5546.382479] Lustre: lustre-OST0000: new disk, initializing [ 5546.384670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5546.387582] Lustre: 404833:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5546.423201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5548.436539] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5551.608526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5551.611522] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5551.645907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5552.035433] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5553.281560] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5553.348836] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5554.618108] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5554.691216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5556.709199] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5556.709330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5556.716505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5556.718944] LustreError: Skipped 1 previous similar message [ 5556.721596] Lustre: Skipped 1 previous similar message [ 5557.406160] Lustre: server umount lustre-OST0000 complete [ 5561.827149] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5561.827337] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.841683] Lustre: Skipped 1 previous similar message [ 5561.846704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5561.850894] Lustre: Skipped 1 previous similar message [ 5564.827425] Lustre: server umount lustre-MDT0000 complete [ 5566.034515] LustreError: 404832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366498 with bad export cookie 8898025015250380173 [ 5566.037513] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5566.037696] LustreError: 404832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5566.162505] Lustre: server umount lustre-MDT0001 complete [ 5569.613356] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5570.268074] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 01:48:23 (1782366503) [ 5572.732721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5572.970272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5574.392736] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5577.556404] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5577.680977] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5579.192355] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5581.935337] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5584.184841] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5587.101236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5587.266185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5589.821455] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5592.847481] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5593.515635] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5599.232843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5602.788842] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5602.789828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5602.796718] Lustre: Skipped 1 previous similar message [ 5602.797847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5602.808397] Lustre: Skipped 1 previous similar message [ 5606.757939] Lustre: server umount lustre-OST0000 complete [ 5608.416286] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5608.419631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5608.424805] Lustre: Skipped 1 previous similar message [ 5608.427429] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.024941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5613.025749] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5614.428690] Lustre: server umount lustre-MDT0000 complete [ 5616.056385] LustreError: 407269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366548 with bad export cookie 8898025015250381041 [ 5616.063594] LustreError: 407269:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5616.064908] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5616.247089] Lustre: server umount lustre-MDT0001 complete [ 5620.818663] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5622.037468] Key type lgssc unregistered [ 5622.155292] LNet: 411009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5622.158637] LNetError: 411009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5622.169339] LNet: Removed LNI 192.168.201.141@tcp [ 5622.492964] Key type .llcrypt unregistered [ 5622.494186] Key type ._llcrypt unregistered [ 5631.061921] Key type ._llcrypt registered [ 5631.062850] Key type .llcrypt registered [ 5631.098959] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5631.192503] Key type .llcrypt unregistered [ 5631.193569] Key type ._llcrypt unregistered [ 5639.406477] Key type ._llcrypt registered [ 5639.407745] Key type .llcrypt registered [ 5639.456434] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5639.877741] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5639.903742] alg: No test for adler32 (adler32-zlib) [ 5640.821560] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5640.959462] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5642.583222] Key type lgssc registered [ 5643.187404] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5646.384875] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5646.393333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5647.582720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5649.173564] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5652.861758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5653.013100] LustreError: 413192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5653.017376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5653.018534] LustreError: 413192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5655.664112] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5657.120284] LustreError: 413196:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.126122] LustreError: 413196:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5657.126399] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5658.143564] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5661.065967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5661.226475] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5662.901796] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5671.394805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5671.396578] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5671.405242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5671.408193] Lustre: Skipped 1 previous similar message [ 5676.516863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5676.520811] Lustre: Skipped 1 previous similar message [ 5676.717982] Lustre: server umount lustre-OST0000 complete [ 5681.123676] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5681.130067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5681.139024] Lustre: Skipped 1 previous similar message [ 5681.143704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5684.595713] Lustre: server umount lustre-MDT0000 complete [ 5686.457642] LustreError: 413177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366619 with bad export cookie 1169108618057412330 [ 5686.463445] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5686.464597] LustreError: 413177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5686.618172] Lustre: server umount lustre-MDT0001 complete [ 5690.687376] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5691.767315] Key type lgssc unregistered [ 5691.894461] LNet: 416344:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5691.896893] LNetError: 416344:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5691.904315] LNet: Removed LNI 192.168.201.141@tcp [ 5692.215420] Key type .llcrypt unregistered [ 5692.216789] Key type ._llcrypt unregistered [ 5703.540193] Key type ._llcrypt registered [ 5703.541055] Key type .llcrypt registered [ 5703.582946] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5703.893631] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5703.923925] alg: No test for adler32 (adler32-zlib) [ 5704.788785] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5704.873903] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5706.463208] Key type lgssc registered [ 5706.992533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5710.421644] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5710.431028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5725.855603] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5730.975358] LustreError: 417711:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.141@tcp: failed processing log, type 4: rc = -110 [ 5757.599168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5760.062683] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5763.429445] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5765.909869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5766.122539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5766.149427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5767.722577] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5771.028736] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5771.162134] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5772.639660] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5775.339547] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5777.927983] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5781.473616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5781.473828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5781.478844] LustreError: Skipped 1 previous similar message [ 5781.490736] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5781.497510] Lustre: Skipped 1 previous similar message [ 5782.497915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5786.597387] Lustre: server umount lustre-OST0000 complete [ 5791.203966] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5791.209299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5791.216923] Lustre: Skipped 1 previous similar message [ 5791.221376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5791.225481] Lustre: Skipped 1 previous similar message [ 5792.736758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5794.643785] Lustre: server umount lustre-MDT0000 complete [ 5795.906201] LustreError: 417718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366728 with bad export cookie 4035201730846343592 [ 5795.913340] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5795.916418] LustreError: 417718:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5796.069896] Lustre: server umount lustre-MDT0001 complete [ 5800.780439] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5801.760305] Key type lgssc unregistered [ 5801.880337] LNet: 421277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5801.882409] LNetError: 421277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5801.889276] LNet: Removed LNI 192.168.201.141@tcp [ 5802.179905] Key type .llcrypt unregistered [ 5802.180994] Key type ._llcrypt unregistered [ 5812.498612] Key type ._llcrypt registered [ 5812.499760] Key type .llcrypt registered [ 5812.541160] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5812.874026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5812.930745] alg: No test for adler32 (adler32-zlib) [ 5813.775166] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5813.856732] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5815.431123] Key type lgssc registered [ 5815.854427] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5819.673676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5819.684836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5820.864426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5822.408571] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5826.102332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5826.245818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5828.020203] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5831.219450] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5833.849245] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5836.751594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5836.904638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5839.321811] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5842.447940] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5845.101522] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5851.875124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5851.879629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5851.886276] Lustre: Skipped 2 previous similar messages [ 5851.890315] Lustre: Skipped 1 previous similar message [ 5855.160051] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.41@tcp (stopping) [ 5856.980826] Lustre: server umount lustre-MDT0000 complete [ 5857.249835] LustreError: 422671:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5857.256026] LustreError: 422671:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5857.984967] LustreError: 422653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.41@tcp arrived at 1782366790 with bad export cookie 5455046404025929277 [ 5857.994194] LustreError: 422653:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5858.902741] LustreError: 422651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782366791 with bad export cookie 5455046404025928633 [ 5858.909836] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5858.909988] LustreError: 422651:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5862.369541] LustreError: 422670:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.370867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5862.381738] LustreError: 422670:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5862.401123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5863.392544] LustreError: 423411:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5863.400475] LustreError: 423411:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5865.079680] Lustre: server umount lustre-OST0000 complete [ 5868.209305] LustreError: 423401:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000d7c5ab6e x1868946993080320/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5868.275030] Lustre: server umount lustre-MDT0001 complete [ 5871.784558] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5872.647735] Key type lgssc unregistered [ 5872.762560] LNet: 426354:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5872.765970] LNetError: 426354:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5872.776376] LNet: Removed LNI 192.168.201.141@tcp [ 5873.062997] Key type .llcrypt unregistered [ 5873.063953] Key type ._llcrypt unregistered [ 5881.929333] Key type ._llcrypt registered [ 5881.931011] Key type .llcrypt registered [ 5881.965039] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5882.306270] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5882.348158] alg: No test for adler32 (adler32-zlib) [ 5883.211301] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5883.303308] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5884.879169] Key type lgssc registered [ 5885.202139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5887.678622] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5887.684979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5888.834577] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5890.286612] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5893.452675] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.620042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5895.143672] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5897.778210] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5899.391211] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5901.688446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5901.876182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5903.908623] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5906.199552] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5913.134301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5918.180884] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5918.184114] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5918.188936] Lustre: Skipped 1 previous similar message [ 5921.720469] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 5921.722551] Lustre: Skipped 1 previous similar message [ 5923.297585] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5924.082151] Lustre: server umount lustre-MDT0001 complete [ 5928.417782] LustreError: 427745:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.418923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.419751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.421431] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5928.421441] Lustre: Skipped 1 previous similar message [ 5928.427068] LustreError: 427745:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5932.326628] Lustre: server umount lustre-OST0000 complete [ 5933.540765] LustreError: 429146:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.300655] Lustre: server umount lustre-MDT0000 complete [ 5940.073494] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 5941.287366] Key type lgssc unregistered [ 5941.407399] LNet: 431444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5941.409691] LNetError: 431444:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5941.420256] LNet: Removed LNI 192.168.201.141@tcp [ 5941.695586] Key type .llcrypt unregistered [ 5941.696561] Key type ._llcrypt unregistered [ 5952.498568] Key type ._llcrypt registered [ 5952.499925] Key type .llcrypt registered [ 5952.545985] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 5959.946726] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5960.324764] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5960.355668] alg: No test for adler32 (adler32-zlib) [ 5961.233782] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 5961.330724] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 5962.919276] Key type lgssc registered [ 5963.551993] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5976.073851] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 5980.750837] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5980.756914] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5981.840477] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5981.849425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5981.886492] Lustre: lustre-MDT0000: new disk, initializing [ 5981.905949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5981.911137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5983.217978] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5988.923218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5989.015798] Lustre: lustre-OST0000: new disk, initializing [ 5989.017427] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5989.018906] Lustre: Skipped 1 previous similar message [ 5989.020713] Lustre: 436449:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5989.043168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5990.734568] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 5994.515175] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5994.518512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5994.528475] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5994.944376] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6000.128181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6000.166880] Lustre: 435431:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6000.191415] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6000.226205] Lustre: lustre-MDT0001: new disk, initializing [ 6000.262067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6000.274395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6000.277258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6001.464863] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6004.752573] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6004.814757] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6005.944915] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6006.004166] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6009.788576] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.41@tcp (stopping) [ 6010.853224] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6010.853327] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6010.861628] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6010.870860] Lustre: Skipped 1 previous similar message [ 6013.554674] Lustre: server umount lustre-MDT0001 complete [ 6021.091660] LustreError: 435466:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6021.097282] LustreError: 435466:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 6026.211207] LustreError: 435436:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6026.221257] LustreError: 435436:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6030.303948] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6030.307408] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.313327] Lustre: Skipped 1 previous similar message [ 6030.316284] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6031.331055] LustreError: 437787:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6032.814686] Lustre: server umount lustre-OST0000 complete [ 6040.957861] Lustre: server umount lustre-MDT0000 complete [ 6047.121060] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6048.546784] Key type lgssc unregistered [ 6048.723707] LNet: 440191:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6048.729438] LNetError: 440191:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6048.743588] LNet: Removed LNI 192.168.201.141@tcp [ 6049.238700] Key type .llcrypt unregistered [ 6049.240929] Key type ._llcrypt unregistered [ 6065.049650] Key type ._llcrypt registered [ 6065.050859] Key type .llcrypt registered [ 6065.089328] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6065.211509] Key type .llcrypt unregistered [ 6065.213087] Key type ._llcrypt unregistered [ 6072.932295] Key type ._llcrypt registered [ 6072.933405] Key type .llcrypt registered [ 6072.971560] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6073.304994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6073.341072] alg: No test for adler32 (adler32-zlib) [ 6074.197882] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 6074.290732] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6075.871316] Key type lgssc registered [ 6076.448480] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6086.937712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6086.961886] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.081276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6088.095884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6088.140339] Lustre: lustre-MDT0000: new disk, initializing [ 6088.175768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6088.185348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6090.059216] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6095.455542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6095.531602] Lustre: 443538:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6095.581561] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6095.586896] Lustre: Skipped 1 previous similar message [ 6095.637776] Lustre: lustre-MDT0001: new disk, initializing [ 6095.701618] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6095.732301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6095.741158] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6097.350473] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6102.289432] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6104.392468] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6107.920947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6108.091434] Lustre: lustre-OST0000: new disk, initializing [ 6108.093553] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6108.096281] Lustre: 445510:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6108.145176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6110.280754] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6113.311065] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6113.317863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6113.374147] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6114.653590] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6118.373709] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.376045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6118.381151] Lustre: Skipped 2 previous similar messages [ 6118.384795] Lustre: Skipped 1 previous similar message [ 6123.488927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6123.492058] Lustre: Skipped 2 previous similar messages [ 6123.695383] Lustre: server umount lustre-MDT0000 complete [ 6125.065771] LustreError: 443532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782367058 with bad export cookie 183196054893868414 [ 6125.072206] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6125.073255] LustreError: 443532:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6125.247328] Lustre: server umount lustre-MDT0001 complete [ 6136.255347] Lustre: server umount lustre-OST0000 complete [ 6141.105825] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 01:57:54 (1782367074) [ 6144.483934] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6144.674943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6146.332876] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6149.991975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6150.112417] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6150.185184] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6151.623269] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6154.022170] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6155.810224] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6171.234479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6171.303646] Lustre: Found index 0 for lustre-OST0000, updating log [ 6171.358423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6173.342712] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6175.761761] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6190.827071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6196.193990] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6196.194537] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6196.196328] LustreError: Skipped 1 previous similar message [ 6196.200571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6196.203312] Lustre: Skipped 1 previous similar message [ 6198.862365] Lustre: server umount lustre-OST0000 complete [ 6211.354708] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6211.355567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6211.360377] Lustre: Skipped 2 previous similar messages [ 6211.363099] Lustre: Skipped 2 previous similar messages [ 6217.492858] Lustre: server umount lustre-MDT0000 complete [ 6218.688549] LustreError: 447515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782367151 with bad export cookie 183196054893869807 [ 6218.691249] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6218.693869] LustreError: 447515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6218.830434] Lustre: server umount lustre-MDT0001 complete [ 6221.748106] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 01:59:15 (1782367155) [ 6235.039285] LNet: 1 local NIs in recovery (showing 1): 192.168.201.141@tcp [ 6806.759127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6806.967922] LustreError: 453974:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.978043] LustreError: 453974:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6806.995826] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6808.867194] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6812.129378] LustreError: 453975:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.753935] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.947881] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6814.626781] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6818.027045] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6820.376715] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6836.710946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6836.857211] Lustre: Found index 0 for lustre-OST0000, updating log [ 6836.969382] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6839.123623] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 6842.427588] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6845.412225] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6845.415364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6845.421312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6845.423828] Lustre: Skipped 3 previous similar messages [ 6849.510571] Lustre: server umount lustre-OST0000 complete [ 6851.303081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6851.304764] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6851.310581] Lustre: Skipped 1 previous similar message [ 6853.344594] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6853.346528] Lustre: Skipped 4 previous similar messages [ 6857.443710] LustreError: 455216:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6857.450028] LustreError: 455216:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6857.462661] Lustre: server umount lustre-MDT0000 complete [ 6859.194883] LustreError: 454710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782367792 with bad export cookie 183196054893870878 [ 6859.199282] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6859.202582] LustreError: 454710:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6859.354720] Lustre: server umount lustre-MDT0001 complete [ 6863.775496] LNet: 1 local NIs in recovery (showing 1): 192.168.201.141@tcp [ 6878.572605] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 6882.956206] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6899.223898] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 02:10:32 (1782367832) [ 6907.015263] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 6908.558200] Key type lgssc unregistered [ 6908.727443] LNet: 460630:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6908.731829] LNetError: 460630:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6969.189371] LNet: Removed LNI 192.168.201.141@tcp [ 6969.483660] Key type .llcrypt unregistered [ 6969.485019] Key type ._llcrypt unregistered [ 6977.760458] Key type ._llcrypt registered [ 6977.762832] Key type .llcrypt registered [ 6977.827719] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 6978.308885] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6978.332930] alg: No test for adler32 (adler32-zlib) [ 6979.193306] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 6979.289372] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 6980.895295] Key type lgssc registered [ 6981.440566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6983.266681] LNet: Added LNI 192.168.201.141@tcp100 [8/256/0/180] [ 6983.633103] LNet: Added LNI 192.168.201.141@tcp101 [8/256/0/180] [ 6984.043975] LNet: Added LNI 192.168.201.141@tcp102 [8/256/0/180] [ 6984.439776] LNet: Added LNI 192.168.201.141@tcp103 [8/256/0/180] [ 6984.837490] LNet: Added LNI 192.168.201.141@tcp104 [8/256/0/180] [ 6985.261747] LNet: Added LNI 192.168.201.141@tcp105 [8/256/0/180] [ 6985.663829] LNet: Added LNI 192.168.201.141@tcp106 [8/256/0/180] [ 6986.064449] LNet: Added LNI 192.168.201.141@tcp107 [8/256/0/180] [ 6986.369354] LNet: Added LNI 192.168.201.141@tcp108 [8/256/0/180] [ 6986.637568] LNet: Added LNI 192.168.201.141@tcp109 [8/256/0/180] [ 6986.908604] LNet: Added LNI 192.168.201.141@tcp110 [8/256/0/180] [ 6987.175717] LNet: Added LNI 192.168.201.141@tcp111 [8/256/0/180] [ 6987.451888] LNet: Added LNI 192.168.201.141@tcp112 [8/256/0/180] [ 6987.831832] LNet: Added LNI 192.168.201.141@tcp113 [8/256/0/180] [ 6988.251048] LNet: Added LNI 192.168.201.141@tcp114 [8/256/0/180] [ 6988.655699] LNet: Added LNI 192.168.201.141@tcp115 [8/256/0/180] [ 6989.051563] LNet: Added LNI 192.168.201.141@tcp116 [8/256/0/180] [ 6989.462790] LNet: Added LNI 192.168.201.141@tcp117 [8/256/0/180] [ 6989.799291] LNet: Added LNI 192.168.201.141@tcp118 [8/256/0/180] [ 6990.078321] LNet: Added LNI 192.168.201.141@tcp119 [8/256/0/180] [ 6990.418862] LNet: Added LNI 192.168.201.141@tcp120 [8/256/0/180] [ 6990.731960] LNet: Added LNI 192.168.201.141@tcp121 [8/256/0/180] [ 6991.047614] LNet: Added LNI 192.168.201.141@tcp122 [8/256/0/180] [ 6991.352220] LNet: Added LNI 192.168.201.141@tcp123 [8/256/0/180] [ 6991.678245] LNet: Added LNI 192.168.201.141@tcp124 [8/256/0/180] [ 6992.028452] LNet: Added LNI 192.168.201.141@tcp125 [8/256/0/180] [ 6992.395324] LNet: Added LNI 192.168.201.141@tcp126 [8/256/0/180] [ 6992.753572] LNet: Added LNI 192.168.201.141@tcp127 [8/256/0/180] [ 6993.158622] LNet: Added LNI 192.168.201.141@tcp128 [8/256/0/180] [ 6993.550412] LNet: Added LNI 192.168.201.141@tcp129 [8/256/0/180] [ 6993.922436] LNet: Added LNI 192.168.201.141@tcp130 [8/256/0/180] [ 6994.335142] LNet: Added LNI 192.168.201.141@tcp131 [8/256/0/180] [ 6994.724258] LNet: Added LNI 192.168.201.141@tcp132 [8/256/0/180] [ 6995.141185] LNet: Added LNI 192.168.201.141@tcp133 [8/256/0/180] [ 6995.570654] LNet: Added LNI 192.168.201.141@tcp134 [8/256/0/180] [ 6995.989526] LNet: Added LNI 192.168.201.141@tcp135 [8/256/0/180] [ 6999.090089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6999.100208] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7000.145913] Lustre: 463626:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7000.229313] LNetError: 463647:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.141 [ 7000.232571] LNetError: Protocol error connecting to 192.168.201.141@tcp118 at host 192.168.201.141:988: is it running a compatible version of Lustre? [ 7000.235637] LNetError: Skipped 1 previous similar message [ 7000.303264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7000.313582] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7000.359291] Lustre: lustre-MDT0000: new disk, initializing [ 7000.383475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7000.389360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7000.800134] LNet: 3 local NIs in recovery (showing 3): 192.168.201.141@tcp129, 192.168.201.141@tcp124, 192.168.201.141@tcp127 [ 7001.313558] LNetError: 463657:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.141 [ 7001.316447] LNetError: 463657:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 7001.318454] LNetError: Protocol error connecting to 192.168.201.141@tcp111 at host 192.168.201.141:988: is it running a compatible version of Lustre? [ 7001.322264] LNetError: Skipped 9 previous similar messages [ 7001.927476] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7003.874199] LNetError: 461091:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.141 [ 7003.880018] LNetError: 461091:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 3 previous similar messages [ 7003.885312] LNetError: Protocol error connecting to 192.168.201.141@tcp124 at host 192.168.201.141:988: is it running a compatible version of Lustre? [ 7003.892132] LNetError: Skipped 3 previous similar messages [ 7006.978888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7007.055853] Lustre: 463642:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7007.099171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7007.101483] Lustre: Skipped 1 previous similar message [ 7007.143199] Lustre: lustre-MDT0001: new disk, initializing [ 7007.159438] Lustre: 464448:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7007.162475] Lustre: 464448:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7007.234085] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7007.257364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7007.263825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7007.968977] LNetError: 463646:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.141 [ 7007.975423] LNetError: Protocol error connecting to 192.168.201.141@tcp124 at host 192.168.201.141:988: is it running a compatible version of Lustre? [ 7009.108812] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7013.650377] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7015.136450] LNetError: 463658:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.141 [ 7015.139764] LNetError: Protocol error connecting to 192.168.201.141@tcp124 at host 192.168.201.141:988: is it running a compatible version of Lustre? [ 7015.871752] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7018.828128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7018.998537] Lustre: lustre-OST0000: new disk, initializing [ 7019.000598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7019.003421] Lustre: 465620:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7019.060151] Lustre: 465625:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7019.065064] Lustre: 465625:0:(lustre_peer.c:105:class_add_uuid()) Skipped 17 previous similar messages [ 7019.095892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7021.668444] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7026.109322] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7028.737611] Lustre: server umount lustre-OST0000 complete [ 7029.247918] Lustre: 466628:0:(lustre_peer.c:105:class_add_uuid()) 192.168.201.141@tcp: can't add NID 192.168.201.141@tcp130: rc = -75 [ 7029.252117] Lustre: 466628:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7030.496676] LNetError: 463661:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.141 [ 7030.502572] LNetError: Protocol error connecting to 192.168.201.141@tcp117 at host 192.168.201.141:988: is it running a compatible version of Lustre? [ 7032.800529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7032.805684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7032.814497] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7034.337514] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7034.338726] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7036.596489] Lustre: server umount lustre-MDT0000 complete [ 7037.938157] LustreError: 463635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782367970 with bad export cookie 7121193119487542897 [ 7037.943238] LustreError: 463635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7037.944460] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7038.076641] Lustre: server umount lustre-MDT0001 complete [ 7041.489155] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7042.879787] Key type lgssc unregistered [ 7043.104584] LNet: 467528:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7043.109942] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7043.114555] LNet: Removed LNI 192.168.201.141@tcp [ 7044.127120] LNet: Removed LNI 192.168.201.141@tcp100 [ 7044.128345] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7044.130161] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7044.132067] LNet: Removed LNI 192.168.201.141@tcp101 [ 7045.151109] LNet: Removed LNI 192.168.201.141@tcp102 [ 7045.152411] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7045.154140] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 7045.156086] LNet: Removed LNI 192.168.201.141@tcp103 [ 7046.175181] LNet: Removed LNI 192.168.201.141@tcp104 [ 7047.199203] LNet: Removed LNI 192.168.201.141@tcp105 [ 7047.201880] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7047.206343] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7047.211187] LNet: Removed LNI 192.168.201.141@tcp106 [ 7048.223118] LNet: Removed LNI 192.168.201.141@tcp107 [ 7049.247281] LNet: Removed LNI 192.168.201.141@tcp108 [ 7050.271094] LNet: Removed LNI 192.168.201.141@tcp109 [ 7051.295103] LNet: Removed LNI 192.168.201.141@tcp110 [ 7051.296256] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7051.298008] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7051.299878] LNet: Removed LNI 192.168.201.141@tcp111 [ 7052.319159] LNet: Removed LNI 192.168.201.141@tcp112 [ 7053.343173] LNet: Removed LNI 192.168.201.141@tcp113 [ 7054.367182] LNet: Removed LNI 192.168.201.141@tcp114 [ 7055.391219] LNet: Removed LNI 192.168.201.141@tcp115 [ 7056.415167] LNet: Removed LNI 192.168.201.141@tcp116 [ 7057.439213] LNet: Removed LNI 192.168.201.141@tcp117 [ 7058.463153] LNet: Removed LNI 192.168.201.141@tcp118 [ 7059.487158] LNet: Removed LNI 192.168.201.141@tcp119 [ 7059.489941] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7059.494555] LNetError: 467528:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7059.499042] LNet: Removed LNI 192.168.201.141@tcp120 [ 7060.511141] LNet: Removed LNI 192.168.201.141@tcp121 [ 7061.535136] LNet: Removed LNI 192.168.201.141@tcp122 [ 7062.559151] LNet: Removed LNI 192.168.201.141@tcp123 [ 7062.560717] LNet: Removed LNI 192.168.201.141@tcp124 [ 7063.583192] LNet: Removed LNI 192.168.201.141@tcp125 [ 7064.607161] LNet: Removed LNI 192.168.201.141@tcp126 [ 7065.631174] LNet: Removed LNI 192.168.201.141@tcp127 [ 7066.655187] LNet: Removed LNI 192.168.201.141@tcp128 [ 7067.679231] LNet: Removed LNI 192.168.201.141@tcp129 [ 7067.681860] LNet: Removed LNI 192.168.201.141@tcp130 [ 7067.684328] LNet: Removed LNI 192.168.201.141@tcp131 [ 7067.686594] LNet: Removed LNI 192.168.201.141@tcp132 [ 7067.688786] LNet: Removed LNI 192.168.201.141@tcp133 [ 7067.691233] LNet: Removed LNI 192.168.201.141@tcp134 [ 7067.700010] LNet: Removed LNI 192.168.201.141@tcp135 [ 7068.124760] Key type .llcrypt unregistered [ 7068.127404] Key type ._llcrypt unregistered [ 7078.105729] Key type ._llcrypt registered [ 7078.107364] Key type .llcrypt registered [ 7078.168547] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7085.596432] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7085.883242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7085.892652] alg: No test for adler32 (adler32-zlib) [ 7086.741698] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 7086.829878] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7088.407167] Key type lgssc registered [ 7088.984983] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7105.948115] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 02:13:59 (1782368039) [ 7112.977904] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7114.025968] Key type lgssc unregistered [ 7114.193533] LNet: 471588:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7114.198096] LNetError: 471588:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7114.209439] LNet: Removed LNI 192.168.201.141@tcp [ 7114.573493] Key type .llcrypt unregistered [ 7114.575175] Key type ._llcrypt unregistered [ 7121.706175] Key type ._llcrypt registered [ 7121.707121] Key type .llcrypt registered [ 7121.743654] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7122.031878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7122.046777] alg: No test for adler32 (adler32-zlib) [ 7122.898978] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 7122.978091] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7124.551156] Key type lgssc registered [ 7124.968333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7128.196080] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7128.201637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7129.279787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7129.288285] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7129.321113] Lustre: lustre-MDT0000: new disk, initializing [ 7129.339895] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7129.344945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7130.662362] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7135.466266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7135.537055] Lustre: 472778:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7135.578593] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7135.582111] Lustre: Skipped 1 previous similar message [ 7135.629503] Lustre: lustre-MDT0001: new disk, initializing [ 7135.659948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7135.672922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7135.675607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7137.037165] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7141.099694] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7143.462463] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7146.410700] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7146.556894] Lustre: lustre-OST0000: new disk, initializing [ 7146.558775] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7146.561614] Lustre: 474746:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7146.598775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7149.064035] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7152.646894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7152.654100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7152.708408] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7153.528729] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7154.675304] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing lnet_if_list [ 7156.951381] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7156.970674] LNet: Added LNI 192.168.201.141@tcp10 [8/256/0/180] [ 7159.236924] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7159.248959] LNet: Added LNI 192.168.201.141@tcp11 [8/256/0/180] [ 7160.927113] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7160.947781] LNet: Added LNI 192.168.201.141@tcp12 [8/256/0/180] [ 7162.414876] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7162.424185] LNet: Added LNI 192.168.201.141@tcp13 [8/256/0/180] [ 7163.880727] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7163.889452] LNet: Added LNI 192.168.201.141@tcp14 [8/256/0/180] [ 7165.289111] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7165.298711] LNet: Added LNI 192.168.201.141@tcp15 [8/256/0/180] [ 7166.678826] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7166.689064] LNet: Added LNI 192.168.201.141@tcp16 [8/256/0/180] [ 7168.081649] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7168.091327] LNet: Added LNI 192.168.201.141@tcp17 [8/256/0/180] [ 7169.611143] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7169.623171] LNet: Added LNI 192.168.201.141@tcp18 [8/256/0/180] [ 7171.360399] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7171.369744] LNet: Added LNI 192.168.201.141@tcp19 [8/256/0/180] [ 7172.864749] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7172.875079] LNet: Added LNI 192.168.201.141@tcp20 [8/256/0/180] [ 7174.453604] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7174.463294] LNet: Added LNI 192.168.201.141@tcp21 [8/256/0/180] [ 7176.089227] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7176.099686] LNet: Added LNI 192.168.201.141@tcp22 [8/256/0/180] [ 7177.759603] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7177.770363] LNet: Added LNI 192.168.201.141@tcp23 [8/256/0/180] [ 7179.534432] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7179.543680] LNet: Added LNI 192.168.201.141@tcp24 [8/256/0/180] [ 7181.292543] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7181.303169] LNet: Added LNI 192.168.201.141@tcp25 [8/256/0/180] [ 7183.030168] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7183.040358] LNet: Added LNI 192.168.201.141@tcp26 [8/256/0/180] [ 7184.703435] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7184.713945] LNet: Added LNI 192.168.201.141@tcp27 [8/256/0/180] [ 7186.590349] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7186.600399] LNet: Added LNI 192.168.201.141@tcp28 [8/256/0/180] [ 7188.476796] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7188.487420] LNet: Added LNI 192.168.201.141@tcp29 [8/256/0/180] [ 7190.411498] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7190.420892] LNet: Added LNI 192.168.201.141@tcp30 [8/256/0/180] [ 7192.301605] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7192.311454] LNet: Added LNI 192.168.201.141@tcp31 [8/256/0/180] [ 7194.247643] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7194.258237] LNet: Added LNI 192.168.201.141@tcp32 [8/256/0/180] [ 7196.233719] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7196.243722] LNet: Added LNI 192.168.201.141@tcp33 [8/256/0/180] [ 7198.229357] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7198.239944] LNet: Added LNI 192.168.201.141@tcp34 [8/256/0/180] [ 7200.376538] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7200.386923] LNet: Added LNI 192.168.201.141@tcp35 [8/256/0/180] [ 7202.481341] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7202.491743] LNet: Added LNI 192.168.201.141@tcp36 [8/256/0/180] [ 7204.666496] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7204.677085] LNet: Added LNI 192.168.201.141@tcp37 [8/256/0/180] [ 7206.817748] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7206.827725] LNet: Added LNI 192.168.201.141@tcp38 [8/256/0/180] [ 7209.160231] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7209.169648] LNet: Added LNI 192.168.201.141@tcp39 [8/256/0/180] [ 7211.419738] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7211.430453] LNet: Added LNI 192.168.201.141@tcp40 [8/256/0/180] [ 7213.801529] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7213.811684] LNet: Added LNI 192.168.201.141@tcp41 [8/256/0/180] [ 7216.175521] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7216.184818] LNet: Added LNI 192.168.201.141@tcp42 [8/256/0/180] [ 7218.523732] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7218.534160] LNet: Added LNI 192.168.201.141@tcp43 [8/256/0/180] [ 7220.891433] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7220.901524] LNet: Added LNI 192.168.201.141@tcp44 [8/256/0/180] [ 7223.346993] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7223.357309] LNet: Added LNI 192.168.201.141@tcp45 [8/256/0/180] [ 7225.851455] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7225.860952] LNet: Added LNI 192.168.201.141@tcp46 [8/256/0/180] [ 7228.418504] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7228.428405] LNet: Added LNI 192.168.201.141@tcp47 [8/256/0/180] [ 7231.065951] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7231.077424] LNet: Added LNI 192.168.201.141@tcp48 [8/256/0/180] [ 7233.632420] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7233.642124] LNet: Added LNI 192.168.201.141@tcp49 [8/256/0/180] [ 7236.294045] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7236.304781] LNet: Added LNI 192.168.201.141@tcp50 [8/256/0/180] [ 7244.770534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7244.772605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7244.774462] Lustre: Skipped 1 previous similar message [ 7249.371742] Lustre: server umount lustre-MDT0000 complete [ 7250.540498] LustreError: 472770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782368183 with bad export cookie 1437704529586033191 [ 7250.544528] LustreError: 472770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7250.544570] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7250.655954] Lustre: server umount lustre-MDT0001 complete [ 7257.298025] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7260.679722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7270.367108] Lustre: 472054:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782368187/real 1782368187] req@00000000dc59f624 x1868948365720960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782368203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7270.374648] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.377587] Lustre: Skipped 1 previous similar message [ 7275.487366] LustreError: 472051:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000661378c7 x1868948365721856/t0(0) o250->MGC192.168.201.141@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7275.579285] LustreError: 474740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7275.584377] LustreError: 474740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7275.590705] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7276.639106] Lustre: 472055:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782368193/real 1782368193] req@000000002ea4b058 x1868948365721088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782368209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7276.784271] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7279.350296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7280.490045] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7280.705499] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7281.631126] Lustre: 472052:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782368198/real 1782368198] req@00000000600967c7 x1868948365721344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782368214 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7282.882726] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7284.473430] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7286.751247] Lustre: 472053:0:(client.c:2489:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1782368203/real 1782368203] req@000000001701eb2c x1868948365721472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1782368219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7294.945674] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7294.948033] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7294.952093] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7294.955124] Lustre: Skipped 2 previous similar messages [ 7295.970436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7295.972416] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7297.833685] Lustre: server umount lustre-OST0000 complete [ 7300.065195] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7300.068618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7300.074306] Lustre: Skipped 1 previous similar message [ 7300.077255] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7300.079679] Lustre: Skipped 1 previous similar message [ 7305.673329] Lustre: server umount lustre-MDT0000 complete [ 7306.208813] LustreError: 498934:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.216090] LustreError: 498934:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7307.235350] LustreError: 472772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782368240 with bad export cookie 1437704529586035200 [ 7307.238531] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7307.243511] LustreError: 472772:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7307.376845] Lustre: server umount lustre-MDT0001 complete [ 7311.879642] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7313.257050] Key type lgssc unregistered [ 7313.466839] LNet: 501704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7313.470774] LNetError: 501704:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7313.474650] LNet: Removed LNI 192.168.201.141@tcp [ 7313.476884] LNet: Removed LNI 192.168.201.141@tcp10 [ 7313.479103] LNet: Removed LNI 192.168.201.141@tcp11 [ 7313.481072] LNet: Removed LNI 192.168.201.141@tcp12 [ 7313.483133] LNet: Removed LNI 192.168.201.141@tcp13 [ 7313.484823] LNet: Removed LNI 192.168.201.141@tcp14 [ 7313.486111] LNet: Removed LNI 192.168.201.141@tcp15 [ 7313.488099] LNet: Removed LNI 192.168.201.141@tcp16 [ 7313.489674] LNet: Removed LNI 192.168.201.141@tcp17 [ 7313.491218] LNet: Removed LNI 192.168.201.141@tcp18 [ 7313.493715] LNet: Removed LNI 192.168.201.141@tcp19 [ 7313.495736] LNet: Removed LNI 192.168.201.141@tcp20 [ 7313.497943] LNet: Removed LNI 192.168.201.141@tcp21 [ 7313.500072] LNet: Removed LNI 192.168.201.141@tcp22 [ 7313.502239] LNet: Removed LNI 192.168.201.141@tcp23 [ 7313.504378] LNet: Removed LNI 192.168.201.141@tcp24 [ 7313.506361] LNet: Removed LNI 192.168.201.141@tcp25 [ 7313.508135] LNet: Removed LNI 192.168.201.141@tcp26 [ 7313.510576] LNet: Removed LNI 192.168.201.141@tcp27 [ 7313.512594] LNet: Removed LNI 192.168.201.141@tcp28 [ 7313.514479] LNet: Removed LNI 192.168.201.141@tcp29 [ 7313.517238] LNet: Removed LNI 192.168.201.141@tcp30 [ 7313.519334] LNet: Removed LNI 192.168.201.141@tcp31 [ 7313.521375] LNet: Removed LNI 192.168.201.141@tcp32 [ 7313.523560] LNet: Removed LNI 192.168.201.141@tcp33 [ 7313.525200] LNet: Removed LNI 192.168.201.141@tcp34 [ 7313.527568] LNet: Removed LNI 192.168.201.141@tcp35 [ 7313.529393] LNet: Removed LNI 192.168.201.141@tcp36 [ 7313.531037] LNet: Removed LNI 192.168.201.141@tcp37 [ 7313.532306] LNet: Removed LNI 192.168.201.141@tcp38 [ 7313.534395] LNet: Removed LNI 192.168.201.141@tcp39 [ 7313.536755] LNet: Removed LNI 192.168.201.141@tcp40 [ 7313.538579] LNet: Removed LNI 192.168.201.141@tcp41 [ 7313.540392] LNet: Removed LNI 192.168.201.141@tcp42 [ 7313.542377] LNet: Removed LNI 192.168.201.141@tcp43 [ 7313.544528] LNet: Removed LNI 192.168.201.141@tcp44 [ 7313.546470] LNet: Removed LNI 192.168.201.141@tcp45 [ 7313.548594] LNet: Removed LNI 192.168.201.141@tcp46 [ 7313.549815] LNet: Removed LNI 192.168.201.141@tcp47 [ 7313.551468] LNet: Removed LNI 192.168.201.141@tcp48 [ 7313.553325] LNet: Removed LNI 192.168.201.141@tcp49 [ 7313.560502] LNet: Removed LNI 192.168.201.141@tcp50 [ 7313.950479] Key type .llcrypt unregistered [ 7313.953139] Key type ._llcrypt unregistered [ 7322.615474] Key type ._llcrypt registered [ 7322.616857] Key type .llcrypt registered [ 7322.662858] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7328.849151] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7329.179252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7329.195071] alg: No test for adler32 (adler32-zlib) [ 7330.054801] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 7330.148560] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7331.735213] Key type lgssc registered [ 7332.119859] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7344.574074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7344.581251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.696053] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7345.713556] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7345.760950] Lustre: lustre-MDT0000: new disk, initializing [ 7345.793406] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7345.802239] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7347.395301] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7351.783231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.830188] Lustre: 504873:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7351.855429] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7351.858759] Lustre: Skipped 1 previous similar message [ 7351.894155] Lustre: lustre-MDT0001: new disk, initializing [ 7351.928612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7351.941552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7351.944773] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7353.402972] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7357.260873] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7359.143407] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7361.706368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.796495] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7361.830355] Lustre: lustre-OST0000: new disk, initializing [ 7361.832863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7361.835517] Lustre: 506888:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7361.867513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7363.864475] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7367.627469] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7369.207989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7369.212491] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7369.246233] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7374.305272] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7374.306910] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7374.307688] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7374.308597] LustreError: Skipped 1 previous similar message [ 7374.312063] Lustre: Skipped 1 previous similar message [ 7376.080240] Lustre: server umount lustre-OST0000 complete [ 7377.887746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7377.889711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7377.893194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7377.895257] Lustre: Skipped 1 previous similar message [ 7379.424203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7379.424720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7383.447955] Lustre: server umount lustre-MDT0000 complete [ 7384.686391] LustreError: 504867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782368317 with bad export cookie 8436538511946541788 [ 7384.689038] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7384.691764] LustreError: 504867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7384.808715] Lustre: server umount lustre-MDT0001 complete [ 7388.251064] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7389.311452] Key type lgssc unregistered [ 7389.438399] LNet: 508793:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7389.441274] LNetError: 508793:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7389.447411] LNet: Removed LNI 192.168.201.141@tcp [ 7389.772155] Key type .llcrypt unregistered [ 7389.773140] Key type ._llcrypt unregistered [ 7397.217595] Key type ._llcrypt registered [ 7397.219103] Key type .llcrypt registered [ 7397.263170] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7397.700705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7397.733673] alg: No test for adler32 (adler32-zlib) [ 7398.607065] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 7398.704048] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7400.279136] Key type lgssc registered [ 7400.648839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7403.175377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7403.180412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7404.313064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7405.656742] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7408.480044] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7408.601494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7409.936247] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7412.148057] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7413.758543] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7415.796705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7415.906108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7417.624674] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7420.074942] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7434.116122] loop0: detected capacity change from 0 to 204800000 [ 7434.220315] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7434.223371] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7434.226736] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7436.235343] loop0: detected capacity change from 0 to 204800000 [ 7436.291846] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7436.297163] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7436.302124] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7437.947199] loop0: detected capacity change from 0 to 204800000 [ 7439.797480] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7439.850491] Lustre: fs15246-OST0000: new disk, initializing [ 7439.852046] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 7439.854333] Lustre: 513655:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7439.871630] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7441.734078] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7445.631111] loop1: detected capacity change from 0 to 204800000 [ 7447.499123] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7447.537678] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 7447.558624] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 7447.607424] Lustre: fs15246-MDT0000: new disk, initializing [ 7447.657627] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7447.666327] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7447.705032] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 7448.735858] Lustre: 514776:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000006e0ee2e4 x1868948655076736/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1782368397 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 7449.148338] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7457.823350] LustreError: 514776:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 [ 7457.826947] LustreError: 514776:0:(fid_request.c:259:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 [ 7457.830953] LustreError: 514776:0:(osp_precreate.c:462:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 [ 7460.834875] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7460.841515] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 7463.061922] Lustre: server umount fs15246-MDT0000 complete [ 7465.954408] LustreError: 510034:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7465.965911] LustreError: 510034:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7466.304733] Lustre: server umount fs15246-OST0000 complete [ 7470.560032] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7470.562499] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7470.567025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7470.568943] Lustre: Skipped 1 previous similar message [ 7476.193463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7476.195187] Lustre: Skipped 2 previous similar messages [ 7481.313155] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7481.315805] Lustre: Skipped 2 previous similar messages [ 7484.383107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7484.435341] Lustre: server umount lustre-OST0000 complete [ 7486.433142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7486.436823] Lustre: Skipped 1 previous similar message [ 7486.438920] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7486.440952] Lustre: Skipped 1 previous similar message [ 7496.672329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7496.674262] Lustre: Skipped 3 previous similar messages [ 7500.255074] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7500.317290] Lustre: server umount lustre-MDT0000 complete [ 7501.545151] LustreError: 514769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782368434 with bad export cookie 14767775080348665889 [ 7501.548298] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.548579] LustreError: 514769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7501.663122] Lustre: server umount lustre-MDT0001 complete [ 7505.256440] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7506.273782] Key type lgssc unregistered [ 7506.394394] LNet: 517867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7506.396652] LNetError: 517867:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7506.405284] LNet: Removed LNI 192.168.201.141@tcp [ 7506.722477] Key type .llcrypt unregistered [ 7506.723472] Key type ._llcrypt unregistered [ 7514.486044] Key type ._llcrypt registered [ 7514.487107] Key type .llcrypt registered [ 7514.525724] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7514.981673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7514.993737] alg: No test for adler32 (adler32-zlib) [ 7515.855889] Lustre: Lustre: Build Version: 2.17.54_83_gdfe0eaa [ 7515.953580] LNet: Added LNI 192.168.201.141@tcp [8/256/0/180] [ 7517.535153] Key type lgssc registered [ 7517.948614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7530.946873] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7534.277623] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7546.867076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7546.872979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7547.970993] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7547.983878] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7548.022678] Lustre: lustre-MDT0000: new disk, initializing [ 7548.044160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7548.050257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7549.495396] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7553.741493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7553.791945] Lustre: 522300:0:(mgs_llog.c:1447:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7553.824512] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7553.827167] Lustre: Skipped 1 previous similar message [ 7553.872812] Lustre: lustre-MDT0001: new disk, initializing [ 7553.912703] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7553.930342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7553.935479] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7555.378649] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7559.122269] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7561.105740] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7563.470233] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7563.586069] Lustre: lustre-OST0000: new disk, initializing [ 7563.587514] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7563.589628] Lustre: 524266:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7563.622176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7565.536786] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_default_debug -1 all [ 7567.227573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7567.230908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7567.261715] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7569.306702] Lustre: DEBUG MARKER: oleg141-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7570.763561] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7570.838232] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7572.310520] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7572.386160] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7577.570828] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7577.571503] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7577.572148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7577.574411] Lustre: Skipped 1 previous similar message [ 7577.579415] LustreError: Skipped 1 previous similar message [ 7582.689760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7582.691668] Lustre: Skipped 2 previous similar messages [ 7587.295132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7587.356810] Lustre: server umount lustre-OST0000 complete [ 7589.855991] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7589.858898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.863307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.928701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7592.929203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7594.721972] Lustre: server umount lustre-MDT0000 complete [ 7595.987359] LustreError: 522294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1782368528 with bad export cookie 17611053306934637732 [ 7595.988295] LustreError: MGC192.168.201.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7595.991885] LustreError: 522294:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7596.111732] Lustre: server umount lustre-MDT0001 complete [ 7604.242624] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing set_hostid [ 7607.468533] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing load_modules_local [ 7618.024730] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 7483 sec ========== 02:22:31 (1782368551) [ 7618.652627] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 02:22:32 (1782368552) === [ 7619.824559] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 02:22:33 (1782368553) === [ 7633.832184] Lustre: DEBUG MARKER: oleg141-server.virtnet: executing unload_modules_local [ 7635.025308] Key type lgssc unregistered [ 7635.182220] LNet: 530900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7635.187232] LNetError: 530900:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7635.195405] LNet: Removed LNI 192.168.201.141@tcp [ 7635.599376] Key type .llcrypt unregistered [ 7635.600719] Key type ._llcrypt unregistered