[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 408657306 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001015] APIC: Switch to symmetric I/O mode setup [ 0.003227] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008756] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010006] pid_max: default: 32768 minimum: 301 [ 0.011133] LSM: Security Framework initializing [ 0.013016] Yama: becoming mindful. [ 0.014027] SELinux: Initializing. [ 0.015061] *** VALIDATE selinux *** [ 0.024172] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028594] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029137] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031061] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.032109] *** VALIDATE tmpfs *** [ 0.034368] *** VALIDATE proc *** [ 0.035135] *** VALIDATE cgroup *** [ 0.036004] *** VALIDATE cgroup2 *** [ 0.037266] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.039133] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.040004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.041022] Spectre V2 : User space: Vulnerable [ 0.042005] Speculative Store Bypass: Vulnerable [ 0.045046] debug: unmapping init [mem 0xffffffff9c659000-0xffffffff9c660fff] [ 0.047199] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.048712] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.049019] ... version: 2 [ 0.050008] ... bit width: 48 [ 0.051006] ... generic registers: 4 [ 0.052008] ... value mask: 0000ffffffffffff [ 0.053006] ... max period: 00007fffffffffff [ 0.054006] ... fixed-purpose events: 3 [ 0.055007] ... event mask: 000000070000000f [ 0.056279] rcu: Hierarchical SRCU implementation. [ 0.058333] smp: Bringing up secondary CPUs ... [ 0.059505] x86: Booting SMP configuration: [ 0.060013] .... node #0, CPUs: #1 #2 #3 [ 0.063119] smp: Brought up 1 node, 4 CPUs [ 0.065007] smpboot: Max logical packages: 1 [ 0.066010] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.277711] node 0 deferred pages initialised in 210ms [ 0.280177] devtmpfs: initialized [ 0.281175] x86/mm: Memory block size: 128MB [ 0.283744] gcov: version magic: 0x41383552 [ 0.285056] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.287074] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.288254] pinctrl core: initialized pinctrl subsystem [ 0.290104] [ 0.290513] ************************************************************* [ 0.293041] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.294009] ** ** [ 0.297008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.298008] ** ** [ 0.300007] ** This means that this kernel is built to expose internal ** [ 0.301006] ** IOMMU data structures, which may compromise security on ** [ 0.302007] ** your system. ** [ 0.304010] ** ** [ 0.306007] ** If you see this message and you are not debugging the ** [ 0.307006] ** kernel, report this immediately to your vendor! ** [ 0.309009] ** ** [ 0.311007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.312006] ************************************************************* [ 0.314711] NET: Registered protocol family 16 [ 0.315368] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.317033] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.319044] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.322135] cpuidle: using governor menu [ 0.323433] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.325053] PCI: Using configuration type 1 for base access [ 0.326150] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.334098] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.336019] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.340069] cryptd: max_cpu_qlen set to 1000 [ 0.342270] ACPI: Added _OSI(Module Device) [ 0.343010] ACPI: Added _OSI(Processor Device) [ 0.344011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.345009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.349361] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.355453] ACPI: Interpreter enabled [ 0.357056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.357990] ACPI: Using IOAPIC for interrupt routing [ 0.359106] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.361384] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.369943] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.371023] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.373009] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.376047] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.379195] acpiphp: Slot [2] registered [ 0.380097] acpiphp: Slot [5] registered [ 0.381126] acpiphp: Slot [6] registered [ 0.383098] acpiphp: Slot [7] registered [ 0.384119] acpiphp: Slot [8] registered [ 0.386104] acpiphp: Slot [9] registered [ 0.388089] acpiphp: Slot [10] registered [ 0.389088] acpiphp: Slot [3] registered [ 0.390087] acpiphp: Slot [4] registered [ 0.391016] acpiphp: Slot [11] registered [ 0.391932] acpiphp: Slot [12] registered [ 0.393069] acpiphp: Slot [13] registered [ 0.393956] acpiphp: Slot [14] registered [ 0.395045] acpiphp: Slot [15] registered [ 0.396042] acpiphp: Slot [16] registered [ 0.396939] acpiphp: Slot [17] registered [ 0.398046] acpiphp: Slot [18] registered [ 0.398937] acpiphp: Slot [19] registered [ 0.399053] acpiphp: Slot [20] registered [ 0.399924] acpiphp: Slot [21] registered [ 0.401060] acpiphp: Slot [22] registered [ 0.401944] acpiphp: Slot [23] registered [ 0.403052] acpiphp: Slot [24] registered [ 0.404015] acpiphp: Slot [25] registered [ 0.405053] acpiphp: Slot [26] registered [ 0.406082] acpiphp: Slot [27] registered [ 0.406962] acpiphp: Slot [28] registered [ 0.408045] acpiphp: Slot [29] registered [ 0.408848] acpiphp: Slot [30] registered [ 0.409061] acpiphp: Slot [31] registered [ 0.409935] PCI host bridge to bus 0000:00 [ 0.411012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.412007] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.413008] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.415009] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.417010] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.419013] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.420155] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.421762] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.423786] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.431009] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.435060] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.437010] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.438007] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.440008] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.441475] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.443618] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.445036] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.447545] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.450983] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.462014] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.465007] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.469055] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.473008] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.476007] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.490016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.501216] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.509016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.520028] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.541017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.547856] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.554012] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.560014] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.569014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.578194] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.582013] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.588013] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.599016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.606000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.614030] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.621014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.636015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.647752] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.653010] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.658014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.673012] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.682106] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.684273] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.685255] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.687226] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.689105] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.693156] iommu: Default domain type: Passthrough [ 0.694447] SCSI subsystem initialized [ 0.696099] ACPI: bus type USB registered [ 0.696918] usbcore: registered new interface driver usbfs [ 0.698027] usbcore: registered new interface driver hub [ 0.699047] usbcore: registered new device driver usb [ 0.700079] pps_core: LinuxPPS API ver. 1 registered [ 0.701005] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.703055] PTP clock support registered [ 0.705157] EDAC MC: Ver: 3.0.0 [ 0.706397] PCI: Using ACPI for IRQ routing [ 0.708579] NetLabel: Initializing [ 0.709005] NetLabel: domain hash size = 128 [ 0.709894] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.711064] NetLabel: unlabeled traffic allowed by default [ 0.712158] vgaarb: loaded [ 0.713271] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.714007] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.719329] clocksource: Switched to clocksource kvm-clock [ 0.797600] VFS: Disk quotas dquot_6.6.0 [ 0.798534] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.800016] *** VALIDATE ramfs *** [ 0.800779] *** VALIDATE hugetlbfs *** [ 0.801996] pnp: PnP ACPI init [ 0.804336] pnp: PnP ACPI: found 6 devices [ 0.816965] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.819566] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.821403] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.823034] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.824612] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.825822] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.827521] NET: Registered protocol family 2 [ 0.829138] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.832860] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.835387] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.838904] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.840970] TCP: Hash tables configured (established 65536 bind 65536) [ 0.843150] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.845331] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.846963] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.848687] NET: Registered protocol family 1 [ 0.850282] RPC: Registered named UNIX socket transport module. [ 0.852447] RPC: Registered udp transport module. [ 0.854227] RPC: Registered tcp transport module. [ 0.855914] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.858253] NET: Registered protocol family 44 [ 0.859554] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.861160] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.862775] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.864646] PCI: CLS 0 bytes, default 64 [ 0.866651] Unpacking initramfs... [ 2.065825] debug: unmapping init [mem 0xffff89ad7cc54000-0xffff89ad7ffbffff] [ 2.070940] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.072792] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.076407] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.501938] Initialise system trusted keyrings [ 2.503072] Key type blacklist registered [ 2.504145] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.510460] zbud: loaded [ 2.512693] *** VALIDATE nfs *** [ 2.513454] *** VALIDATE nfs4 *** [ 2.514362] pstore: using deflate compression [ 2.516031] Platform Keyring initialized [ 2.583774] NET: Registered protocol family 38 [ 2.585082] Key type asymmetric registered [ 2.586260] Asymmetric key parser 'x509' registered [ 2.587639] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.589787] io scheduler mq-deadline registered [ 2.591069] io scheduler kyber registered [ 2.592437] io scheduler bfq registered [ 2.593715] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.595794] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.597851] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.599976] ACPI: Power Button [PWRF] [ 2.604615] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.610668] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.622125] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.627910] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.643162] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.669141] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 2.695921] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 2.700463] Non-volatile memory driver v1.3 [ 2.701826] Linux agpgart interface v0.103 [ 2.728601] virtio_blk virtio1: [vda] 145912 512-byte logical blocks (74.7 MB/71.2 MiB) [ 2.730822] vda: detected capacity change from 0 to 74706944 [ 2.744462] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 2.746893] vdb: detected capacity change from 0 to 1073741824 [ 2.759153] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.761371] vdc: detected capacity change from 0 to 2621440000 [ 2.777698] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 2.780385] vdd: detected capacity change from 0 to 2621440000 [ 2.793743] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.796335] vde: detected capacity change from 0 to 4294967296 [ 2.809466] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 2.812301] vdf: detected capacity change from 0 to 4294967296 [ 2.818644] libphy: Fixed MDIO Bus: probed [ 2.824692] usbcore: registered new interface driver usbserial_generic [ 2.826649] usbserial: USB Serial support registered for generic [ 2.828589] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 2.832187] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 2.833430] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 2.835261] mousedev: PS/2 mouse device common for all mice [ 2.837120] rtc_cmos 00:05: RTC can wake from S4 [ 2.839425] rtc_cmos 00:05: registered as rtc0 [ 2.840739] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 2.842472] intel_pstate: CPU model not supported [ 2.843760] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 2.846355] hid: raw HID events driver (C) Jiri Kosina [ 2.848687] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 2.848751] usbcore: registered new interface driver usbhid [ 2.852625] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 2.852929] usbhid: USB HID core driver [ 2.856857] drop_monitor: Initializing network drop monitor service [ 2.858971] Initializing XFRM netlink socket [ 2.860135] NET: Registered protocol family 10 [ 2.862202] Segment Routing with IPv6 [ 2.863327] NET: Registered protocol family 17 [ 2.864863] mpls_gso: MPLS GSO support [ 2.869965] RAS: Correctable Errors collector initialized. [ 2.871644] AVX version of gcm_enc/dec engaged. [ 2.872850] AES CTR mode by8 optimization enabled [ 2.943475] sched_clock: Marking stable (2943450092, 0)->(3692291306, -748841214) [ 2.946789] registered taskstats version 1 [ 2.950158] Loading compiled-in X.509 certificates [ 2.952128] zswap: loaded using pool lzo/zbud [ 2.970721] Key type big_key registered [ 2.983493] Key type encrypted registered [ 2.984878] ima: No TPM chip found, activating TPM-bypass! [ 2.990343] ima: Allocated hash algorithm: sha1 [ 2.991786] ima: No architecture policies found [ 2.993412] evm: Initialising EVM extended attributes: [ 2.996387] evm: security.selinux [ 2.997551] evm: security.ima [ 2.998552] evm: security.capability [ 2.999906] evm: HMAC attrs: 0x1 [ 3.002302] rtc_cmos 00:05: setting system clock to 2026-08-17 02:38:26 UTC (1786934306) [ 3.007340] debug: unmapping init [mem 0xffffffff9d603000-0xffffffff9d7fffff] [ 3.009684] debug: unmapping init [mem 0xffffffff9c382000-0xffffffff9c658fff] [ 3.019101] Write protecting the kernel read-only data: 28672k [ 3.021312] debug: unmapping init [mem 0xffffffff9aa03000-0xffffffff9abfffff] [ 3.023096] debug: unmapping init [mem 0xffffffff9b314000-0xffffffff9b3fffff] [ 3.058668] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.074789] systemd[1]: Detected virtualization kvm. [ 3.077904] systemd[1]: Detected architecture x86-64. [ 3.080866] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.120460] systemd[1]: No hostname configured. [ 3.122320] systemd[1]: Set hostname to . [ 3.124369] random: systemd: uninitialized urandom read (16 bytes read) [ 3.126414] systemd[1]: Initializing machine ID from random generator. [ 3.174897] random: ln: uninitialized urandom read (6 bytes read) [ 3.257371] random: systemd: uninitialized urandom read (16 bytes read) [ 3.259435] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.263132] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.266962] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Sockets. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 3.889418] device-mapper: uevent: version 1.0.3 [ 3.890894] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ 4.618976] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.665612] random: fast init done [ 4.718974] scsi host0: ata_piix [ 4.737251] scsi host1: ata_piix [ 4.739074] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.742200] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.515381] random: crng init done [ 9.516622] random: 7 urandom warning(s) missed due to ratelimiting [ 9.857386] dracut-initqueue[581]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 11.101466] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Swap. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. Stopping udev Kernel Device Manager... [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.393637] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.744888] SELinux: Disabled at runtime. [ 12.814553] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.822329] systemd[1]: Detected virtualization kvm. [ 12.824294] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.539403] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.544348] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.552229] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.555286] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.558621] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.566384] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.574772] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Apply Kernel Variables... [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ 13.860717] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 14.518511] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 15.111848] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 15.122525] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 15.501818] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 15.570873] EDAC sbridge: Ver: 1.1.2 [ 18.086370] Key type dns_resolver registered [ 18.429344] NFS: Registering the id_resolver key type [ 18.431803] Key type id_resolver registered [ 18.433302] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Mark the need to relabel after reboot. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Restore /run/initramfs on shutdown... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Login Service. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. [ OK ] Started Authorization Manager. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg109-server login: [ 53.235114] libcfs: loading out-of-tree module taints kernel. [ 53.266179] Key type ._llcrypt registered [ 53.269395] Key type .llcrypt registered [ 53.330463] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 62.147106] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 62.758065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 62.767297] alg: No test for adler32 (adler32-zlib) [ 63.751523] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 64.045484] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 65.647215] Key type lgssc registered [ 66.261701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.352299] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.092975] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 90.403354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 90.411151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 91.490351] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 91.499752] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 91.535442] Lustre: lustre-MDT0000: new disk, initializing [ 91.556880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 91.564290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 92.839786] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 97.938102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 97.974215] Lustre: 6477:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 97.987265] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 97.989671] Lustre: Skipped 1 previous similar message [ 98.028137] Lustre: lustre-MDT0001: new disk, initializing [ 98.052388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 98.061704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 98.065757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 99.426076] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.684866] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 104.889691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 104.975327] Lustre: lustre-OST0000: new disk, initializing [ 104.976803] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 104.979246] Lustre: 8415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 104.997484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 106.875096] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 110.057196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 110.060150] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 110.076910] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 112.068228] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 112.106106] Lustre: lustre-OST0001: new disk, initializing [ 112.108310] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 112.111240] Lustre: 9489:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 112.126908] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 113.935045] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 119.276899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 119.279970] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 119.292826] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 119.789790] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 126.448696] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 128.334677] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing check_logdir /tmp/testlogs/ [ 129.709445] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing yml_node [ 131.086914] Lustre: DEBUG MARKER: Client: 2.17.57.1 [ 131.885627] Lustre: DEBUG MARKER: MDS: 2.17.57.1 [ 132.716418] Lustre: DEBUG MARKER: OSS: 2.17.57.1 [ 133.225256] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sun Aug 16 22:40:35 EDT 2026 [ 138.444021] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 138.897534] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 154.591517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 154.593909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 154.597088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 155.103506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 155.103926] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 155.105331] Lustre: Skipped 2 previous similar messages [ 155.110628] Lustre: Skipped 2 previous similar messages [ 156.699731] Lustre: server umount lustre-MDT0000 complete [ 157.851928] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934461 with bad export cookie 11788438651239724429 [ 157.853466] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 157.855724] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 160.223705] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 160.226811] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 160.228596] Lustre: Skipped 1 previous similar message [ 160.233010] Lustre: Skipped 2 previous similar messages [ 162.272018] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 162.274786] Lustre: Skipped 1 previous similar message [ 163.997296] Lustre: server umount lustre-MDT0001 complete [ 171.321974] Lustre: server umount lustre-OST0000 complete [ 178.633381] Lustre: server umount lustre-OST0001 complete [ 180.637994] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 183.697620] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 195.944511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 196.034987] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 196.045600] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 196.079612] Lustre: lustre-MDT0000: new disk, initializing [ 196.100309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 196.107504] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 197.547423] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 201.852337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 201.896655] Lustre: 15028:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 201.996063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 203.333224] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 207.102835] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 209.098087] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 211.384428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 211.482897] Lustre: lustre-OST0000: new disk, initializing [ 211.484413] Lustre: Skipped 1 previous similar message [ 211.485989] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 211.488383] Lustre: Skipped 2 previous similar messages [ 211.490508] Lustre: 16998:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 211.519078] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 211.521225] Lustre: Skipped 1 previous similar message [ 213.240627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 213.243375] Lustre: Skipped 1 previous similar message [ 213.245293] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 213.286393] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 213.525335] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 217.454211] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 218.921395] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 218.989200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 220.415512] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 220.492240] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 223.713124] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 223.714652] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 223.716088] LustreError: Skipped 1 previous similar message [ 223.721225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 223.724122] Lustre: Skipped 1 previous similar message [ 227.215326] Lustre: server umount lustre-OST0000 complete [ 228.832317] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 228.836235] Lustre: Skipped 2 previous similar messages [ 233.952053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 233.954865] Lustre: Skipped 3 previous similar messages [ 234.588454] Lustre: server umount lustre-MDT0000 complete [ 235.900101] LustreError: 15023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934539 with bad export cookie 11788438651239725948 [ 235.903825] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 235.904043] LustreError: 15023:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 236.018617] Lustre: server umount lustre-MDT0001 complete [ 237.483925] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 22:42:19 (1786934539) [ 238.327769] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 238.877583] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 22:42:21 (1786934541) [ 241.184326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 241.341338] LustreError: 19405:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 241.348081] LustreError: 19405:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 241.366813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 242.869366] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 245.990887] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 247.631186] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 250.245451] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 252.121885] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 254.574557] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 256.774660] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 259.407419] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 276.449875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 276.452675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 276.454026] Lustre: Skipped 1 previous similar message [ 276.454177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 276.456434] LustreError: Skipped 1 previous similar message [ 276.458197] Lustre: Skipped 1 previous similar message [ 278.741906] Lustre: server umount lustre-OST0000 complete [ 294.367121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 294.429357] Lustre: server umount lustre-MDT0000 complete [ 295.720454] LustreError: 19386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934599 with bad export cookie 11788438651239726830 [ 295.723142] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 295.724148] LustreError: 19386:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 295.854418] Lustre: server umount lustre-MDT0001 complete [ 299.589359] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 300.670054] Key type lgssc unregistered [ 300.801415] LNet: 22960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 300.803934] LNetError: 22960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 300.812308] LNet: Removed LNI 192.168.201.109@tcp [ 301.172128] Key type .llcrypt unregistered [ 301.173572] Key type ._llcrypt unregistered [ 310.100604] Key type ._llcrypt registered [ 310.101903] Key type .llcrypt registered [ 310.140274] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 310.516748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 310.559572] alg: No test for adler32 (adler32-zlib) [ 311.413230] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 311.502563] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 313.087172] Key type lgssc registered [ 313.449174] Lustre: Echo OBD driver; http://www.lustre.org/ [ 316.243088] loop: module loaded [ 316.269980] loop0: detected capacity change from 0 to 204800000 [ 316.332191] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 316.336639] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 316.459423] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.504241] loop0: detected capacity change from 0 to 204800000 [ 318.569436] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.574510] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 318.579339] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 320.295500] loop0: detected capacity change from 0 to 204800000 [ 322.246468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 322.251988] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 323.341689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 323.357316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 323.399655] Lustre: lustre-MDT0000: new disk, initializing [ 323.429212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 323.437687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 324.796234] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 328.533754] loop1: detected capacity change from 0 to 204800000 [ 330.505818] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 330.621490] Lustre: lustre-OST07c6: new disk, initializing [ 330.623851] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 330.626443] Lustre: Skipped 1 previous similar message [ 330.629269] Lustre: 26120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 330.655914] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 332.359387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 332.362976] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 332.375794] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 332.736131] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 334.936601] Lustre: 25814:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 334.939256] Lustre: 25814:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 334.941338] Lustre: 25814:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 334.944270] Lustre: 25814:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 334.946576] Lustre: 25814:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 334.949658] Lustre: 25814:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 352.237913] Lustre: 25106:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 352.241014] Lustre: 25106:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 352.243846] Lustre: 25106:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 352.246384] Lustre: 25106:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 352.249919] Lustre: 25106:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 352.252746] Lustre: 25106:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 355.787281] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 355.790645] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 355.793433] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 355.796029] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 355.798979] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 355.802544] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 355.804546] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 355.806807] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 355.808796] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 355.810978] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 355.813756] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 355.815859] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 358.872319] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 358.878510] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 358.881476] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 358.883660] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 358.886134] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 358.888534] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 358.891035] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 358.893869] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 358.897292] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 358.900947] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 358.904165] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 358.907417] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 365.487800] Lustre: 25814:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 365.491762] Lustre: 25814:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 365.495903] Lustre: 25814:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 365.499198] Lustre: 25814:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 365.502529] Lustre: 25814:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 365.504904] Lustre: 25814:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 365.507669] Lustre: 25814:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 365.510729] Lustre: 25814:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 365.512939] Lustre: 25814:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 365.515404] Lustre: 25814:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 365.517780] Lustre: 25814:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 365.519908] Lustre: 25814:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 375.064136] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 375.067033] Lustre: 25108:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 375.069677] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 375.071361] Lustre: 25108:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 375.074311] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 375.076854] Lustre: 25108:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 375.079016] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 375.081795] Lustre: 25108:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 375.084609] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 375.087284] Lustre: 25108:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 375.089445] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 375.091657] Lustre: 25108:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 419.297071] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 419.302444] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 419.302755] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 421.982938] Lustre: server umount lustre-OST07c6 complete [ 425.421322] Lustre: server umount lustre-MDT0000 complete [ 431.198364] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 432.237325] Key type lgssc unregistered [ 432.363399] LNet: 28697:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 432.366112] LNetError: 28697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 432.375353] LNet: Removed LNI 192.168.201.109@tcp [ 432.681915] Key type .llcrypt unregistered [ 432.683423] Key type ._llcrypt unregistered [ 440.145017] Key type ._llcrypt registered [ 440.146386] Key type .llcrypt registered [ 440.186328] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 440.519392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 440.544171] alg: No test for adler32 (adler32-zlib) [ 441.410332] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 441.496411] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 443.079166] Key type lgssc registered [ 443.439824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 446.081687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 446.089047] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 447.234213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 448.697778] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 451.750263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 451.890693] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 453.349229] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 455.905299] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 457.772126] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 460.176659] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 460.293720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 462.172147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 464.863305] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 471.590356] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 481.760596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.762712] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.763949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.768692] Lustre: Skipped 1 previous similar message [ 481.771857] Lustre: Skipped 1 previous similar message [ 486.881130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 486.883019] Lustre: Skipped 1 previous similar message [ 487.113932] Lustre: server umount lustre-OST0000 complete [ 492.000591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.005267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 492.007394] Lustre: Skipped 1 previous similar message [ 494.488140] Lustre: server umount lustre-MDT0000 complete [ 495.693282] LustreError: 29926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934799 with bad export cookie 6957069271834812014 [ 495.696730] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 495.697443] LustreError: 29926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 495.813026] Lustre: server umount lustre-MDT0001 complete [ 499.507314] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 500.579642] Key type lgssc unregistered [ 500.703430] LNet: 33700:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 500.706292] LNetError: 33700:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 500.715348] LNet: Removed LNI 192.168.201.109@tcp [ 501.044694] Key type .llcrypt unregistered [ 501.046247] Key type ._llcrypt unregistered [ 508.462327] Key type ._llcrypt registered [ 508.463825] Key type .llcrypt registered [ 508.511163] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 508.878949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 508.929695] alg: No test for adler32 (adler32-zlib) [ 509.782548] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 509.868192] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 511.447164] Key type lgssc registered [ 511.799108] Lustre: Echo OBD driver; http://www.lustre.org/ [ 514.233463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 514.239915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 515.378352] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 516.617755] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 519.335928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 519.456574] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 520.766911] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 523.241202] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 525.021855] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 527.192902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 527.299830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.105336] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 531.451749] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 539.624434] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 549.859871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 549.861637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 549.862377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 549.862475] LustreError: Skipped 1 previous similar message [ 549.865584] Lustre: Skipped 1 previous similar message [ 551.500334] Lustre: server umount lustre-OST0000 complete [ 554.976757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 554.977565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 554.981469] Lustre: Skipped 1 previous similar message [ 554.983172] Lustre: Skipped 1 previous similar message [ 560.096086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 560.097892] Lustre: Skipped 2 previous similar messages [ 565.215915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 565.218308] Lustre: Skipped 1 previous similar message [ 567.263149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 567.323577] Lustre: server umount lustre-MDT0000 complete [ 568.555409] LustreError: 36787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934872 with bad export cookie 12914651895163944375 [ 568.558793] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 568.559138] LustreError: 36787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 568.669284] Lustre: server umount lustre-MDT0001 complete [ 572.023517] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 573.027944] Key type lgssc unregistered [ 573.142428] LNet: 38501:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 573.144845] LNetError: 38501:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 573.153286] LNet: Removed LNI 192.168.201.109@tcp [ 573.445464] Key type .llcrypt unregistered [ 573.447043] Key type ._llcrypt unregistered [ 582.063906] Key type ._llcrypt registered [ 582.065120] Key type .llcrypt registered [ 582.099949] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 582.419817] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 582.450614] alg: No test for adler32 (adler32-zlib) [ 583.294099] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 583.381062] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 584.959130] Key type lgssc registered [ 585.281657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 587.714470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 587.720766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 588.857295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 590.083907] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 592.807925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 592.922985] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 594.290740] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 596.634135] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 598.405155] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 600.633933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 600.751088] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 602.581783] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 605.015177] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 614.054275] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 618.464536] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 618.467457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 618.472158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 619.488315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 619.490172] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 619.495205] Lustre: Skipped 1 previous similar message [ 620.571564] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 620.573523] Lustre: Skipped 2 previous similar messages [ 623.762925] Lustre: server umount lustre-MDT0000 complete [ 624.608667] LustreError: 39913:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 624.613806] LustreError: 39913:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 625.003441] LustreError: 39876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786934928 with bad export cookie 707846582747299671 [ 625.006020] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 625.007316] LustreError: 39876:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 625.682248] LustreError: 41326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 625.689694] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 629.727857] LustreError: 39891:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 629.728177] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 629.738367] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 631.142788] Lustre: server umount lustre-MDT0001 complete [ 657.375107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 657.431093] Lustre: server umount lustre-OST0000 complete [ 662.742904] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 663.667950] Key type lgssc unregistered [ 663.794356] LNet: 43775:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 663.797561] LNetError: 43775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 663.807273] LNet: Removed LNI 192.168.201.109@tcp [ 664.095048] Key type .llcrypt unregistered [ 664.096507] Key type ._llcrypt unregistered [ 672.577564] Key type ._llcrypt registered [ 672.579183] Key type .llcrypt registered [ 672.619093] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 673.131559] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 673.173069] alg: No test for adler32 (adler32-zlib) [ 674.025285] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 674.109900] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 675.687128] Key type lgssc registered [ 676.011786] Lustre: Echo OBD driver; http://www.lustre.org/ [ 678.410299] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 678.416060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 679.548803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 680.787529] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 683.544856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 683.668566] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 685.009780] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 687.370160] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 689.106275] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 691.249987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 691.352150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 693.116920] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 695.396417] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 704.677471] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 710.113227] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 710.113524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 710.116459] LustreError: Skipped 1 previous similar message [ 710.120642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 710.123304] Lustre: Skipped 1 previous similar message [ 711.193139] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 714.376822] Lustre: server umount lustre-OST0000 complete [ 727.008146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 727.011972] Lustre: Skipped 1 previous similar message [ 727.014947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 727.016912] Lustre: Skipped 1 previous similar message [ 730.080843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 730.082376] Lustre: Skipped 1 previous similar message [ 735.200256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 735.202522] Lustre: Skipped 1 previous similar message [ 741.343122] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 741.403739] Lustre: server umount lustre-MDT0000 complete [ 742.612147] LustreError: 45904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935046 with bad export cookie 3782986019128299773 [ 742.614783] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.615799] LustreError: 45904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 748.788532] Lustre: server umount lustre-MDT0001 complete [ 752.263617] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 753.204948] Key type lgssc unregistered [ 753.325316] LNet: 48899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 753.327575] LNetError: 48899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 753.339283] LNet: Removed LNI 192.168.201.109@tcp [ 753.620876] Key type .llcrypt unregistered [ 753.621975] Key type ._llcrypt unregistered [ 762.048981] Key type ._llcrypt registered [ 762.050359] Key type .llcrypt registered [ 762.087803] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 762.473946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 762.483056] alg: No test for adler32 (adler32-zlib) [ 763.326795] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 763.409482] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 764.983132] Key type lgssc registered [ 765.309270] Lustre: Echo OBD driver; http://www.lustre.org/ [ 767.699660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 767.705956] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 768.838962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 770.141875] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 772.783892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 772.898359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 774.130175] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 776.403374] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 778.068182] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 780.159216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 780.277367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.084577] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 784.337402] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 793.575075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 796.710067] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 797.772605] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 808.458314] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-1622h51m50s [ 808.956760] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 810.008561] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 811.025394] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 814.049465] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.051038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 814.054487] Lustre: Skipped 2 previous similar messages [ 814.056754] Lustre: Skipped 1 previous similar message [ 814.060395] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 815.830901] Lustre: server umount lustre-MDT0000 complete [ 816.926729] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 817.169245] LustreError: 52652:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 817.173846] LustreError: 52652:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 819.151900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.192418] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.278764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.297109] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 820.325840] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 820.566752] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 821.690599] Lustre: DEBUG MARKER: Wait for df (36626) ... [ 822.201822] Lustre: DEBUG MARKER: done [ 824.288188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 824.288473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 824.290746] LustreError: Skipped 1 previous similar message [ 824.291239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 824.294947] Lustre: Skipped 1 previous similar message [ 824.296296] Lustre: Skipped 4 previous similar messages [ 825.232186] Lustre: server umount lustre-OST0000 complete [ 828.566123] Lustre: server umount lustre-MDT0000 complete [ 829.819299] LustreError: 52937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935133 with bad export cookie 2259173263786539030 [ 829.823156] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 829.824208] LustreError: 52937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 836.005772] Lustre: server umount lustre-MDT0001 complete [ 839.364159] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 840.334023] Key type lgssc unregistered [ 840.464326] LNet: 55227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 840.466295] LNetError: 55227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 840.473260] LNet: Removed LNI 192.168.201.109@tcp [ 840.783143] Key type .llcrypt unregistered [ 840.784507] Key type ._llcrypt unregistered [ 854.551560] Key type ._llcrypt registered [ 854.552593] Key type .llcrypt registered [ 854.585695] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 854.889583] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 854.930694] alg: No test for adler32 (adler32-zlib) [ 855.780183] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 855.857554] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 857.431131] Key type lgssc registered [ 857.783407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 860.245493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 860.250928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 861.328648] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 861.337040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 861.398720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 862.768798] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 865.590842] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 865.621287] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 865.630919] Lustre: 57411:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 865.727942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 866.987787] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 869.316437] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 871.113626] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 873.299951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 873.375329] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 873.452478] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 875.264586] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 877.625814] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 878.603446] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 886.453081] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-1622h53m08s [ 886.994125] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 889.243647] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 890.048072] Lustre: DEBUG MARKER: Creating a test file and stat it [ 890.063427] LustreError: 59927:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 910.366808] Lustre: lustre-MDT0000: Client bf9f144a-7e53-4530-ad71-2a269a99ab68 (at 192.168.201.9@tcp) reconnecting [ 930.103139] LustreError: 59927:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 930.754093] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 931.483488] Lustre: DEBUG MARKER: done [ 934.881839] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 934.882159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 934.883680] LustreError: Skipped 1 previous similar message [ 934.887514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 934.889879] Lustre: Skipped 1 previous similar message [ 939.019440] Lustre: server umount lustre-OST0000 complete [ 942.559627] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 942.561510] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 942.564526] Lustre: Skipped 1 previous similar message [ 942.565976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.584478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 943.585984] Lustre: Skipped 2 previous similar messages [ 945.632494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 945.635170] Lustre: Skipped 1 previous similar message [ 946.323573] Lustre: server umount lustre-MDT0000 complete [ 947.493886] LustreError: 58152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935250 with bad export cookie 5691683758186841199 [ 947.496124] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 947.496981] LustreError: 58152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 947.610542] Lustre: server umount lustre-MDT0001 complete [ 951.162244] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 952.191954] Key type lgssc unregistered [ 952.321456] LNet: 61734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 952.324534] LNetError: 61734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 952.333326] LNet: Removed LNI 192.168.201.109@tcp [ 952.625985] Key type .llcrypt unregistered [ 952.627055] Key type ._llcrypt unregistered [ 967.572819] Key type ._llcrypt registered [ 967.573892] Key type .llcrypt registered [ 967.613607] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 967.948156] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 967.988060] alg: No test for adler32 (adler32-zlib) [ 968.855307] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 968.946589] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 970.535169] Key type lgssc registered [ 970.904609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 971.878194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 971.883178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 972.956855] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 972.963290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 973.015510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 973.340426] Lustre: Failing over lustre-MDT0000 [ 973.437241] Lustre: server umount lustre-MDT0000 complete [ 975.464304] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 22:54:37 (1786935277) [ 977.539152] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 977.670362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 978.973139] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 981.633096] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.661485] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 981.671048] Lustre: 64509:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 981.760667] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 983.019873] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 985.203121] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 986.851775] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 988.935822] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 989.020268] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 989.079099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.949190] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 993.341230] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 993.806779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1001.783821] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1004.001780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1004.003510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1004.004799] Lustre: Skipped 1 previous similar message [ 1004.007643] Lustre: Skipped 2 previous similar messages [ 1009.121475] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1009.124300] Lustre: Skipped 2 previous similar messages [ 1009.565945] Lustre: server umount lustre-MDT0000 complete [ 1010.867166] LustreError: 64540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935314 with bad export cookie 15677778318465915402 [ 1010.870228] LustreError: 64540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1010.871308] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.995765] Lustre: server umount lustre-MDT0001 complete [ 1012.158695] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1013.749344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1016.682580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1021.407700] LustreError: 63431:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004c8b349b x1873737439804544/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1021.547150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1021.580879] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1022.871731] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1025.762441] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1025.867095] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.871614] Lustre: Skipped 1 previous similar message [ 1027.293115] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1029.776575] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1029.983134] Lustre: 63434:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786935317/real 1786935317] req@00000000ae0a2528 x1873737439804160/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786935333 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1031.007142] Lustre: 63435:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786935318/real 1786935318] req@00000000f55214a5 x1873737439804288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786935334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1031.141719] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1031.646954] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1034.465237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.465744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1034.466806] Lustre: Skipped 1 previous similar message [ 1034.469884] Lustre: Skipped 1 previous similar message [ 1035.487221] Lustre: 63432:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786935322/real 1786935322] req@0000000028a74e59 x1873737439804416/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786935338 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1037.535206] Lustre: 63433:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786935324/real 1786935324] req@00000000a0a1a60f x1873737439804672/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786935340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1039.579880] Lustre: server umount lustre-MDT0000 complete [ 1039.842525] LustreError: 68278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.848192] LustreError: 68278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 1040.838285] LustreError: 64501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935344 with bad export cookie 15677778318465917670 [ 1040.840992] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.841109] LustreError: 64501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1040.948597] Lustre: server umount lustre-MDT0001 complete [ 1043.006880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1045.997666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.103697] LustreError: 63431:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000004aac0b18 x1873737439823360/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1051.230143] LustreError: 69903:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1051.252535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1051.254593] Lustre: Skipped 1 previous similar message [ 1051.274378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1052.484736] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1055.137429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1055.228134] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1056.546809] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1058.745482] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1060.324274] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1060.490939] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1061.343167] Lustre: 63433:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786935348/real 1786935348] req@00000000984f7cd5 x1873737439822976/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786935364 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1065.442357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1065.442487] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1065.443141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1065.443146] Lustre: Skipped 1 previous similar message [ 1065.451374] Lustre: Skipped 1 previous similar message [ 1068.185194] Lustre: server umount lustre-MDT0000 complete [ 1069.454508] LustreError: 66362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935372 with bad export cookie 15677778318465919042 [ 1069.457361] LustreError: 66362:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1069.458127] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1069.575863] Lustre: server umount lustre-MDT0001 complete [ 1071.764307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1072.942026] Lustre: DEBUG MARKER: files compared the same [ 1083.946311] Lustre: server umount lustre-OST0000 complete [ 1089.428833] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1090.554850] Key type lgssc unregistered [ 1090.698408] LNet: 74229:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1090.700641] LNetError: 74229:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1090.711396] LNet: Removed LNI 192.168.201.109@tcp [ 1091.027756] Key type .llcrypt unregistered [ 1091.028706] Key type ._llcrypt unregistered [ 1100.184941] Key type ._llcrypt registered [ 1100.186358] Key type .llcrypt registered [ 1100.233247] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1100.594424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1100.669625] alg: No test for adler32 (adler32-zlib) [ 1101.531665] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1101.623059] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1103.207145] Key type lgssc registered [ 1103.585208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1106.136181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1106.142266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.293862] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1108.692599] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 1111.465910] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1111.567295] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1112.952109] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 1115.419947] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1117.307072] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1119.576400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1119.677720] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1121.640349] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 1123.993609] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1133.026068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1137.119474] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1137.122569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1137.128082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1138.144098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1138.144270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.150945] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1143.179655] Lustre: server umount lustre-OST0000 complete [ 1147.359491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1147.361625] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1147.364897] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1147.367276] Lustre: Skipped 1 previous similar message [ 1150.543724] Lustre: server umount lustre-MDT0000 complete [ 1151.780740] LustreError: 75604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935455 with bad export cookie 17737013480856621805 [ 1151.782228] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1151.787193] LustreError: 75604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1151.893243] Lustre: server umount lustre-MDT0001 complete [ 1155.633541] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1156.728494] Key type lgssc unregistered [ 1156.872326] LNet: 79177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1156.874559] LNetError: 79177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1156.884296] LNet: Removed LNI 192.168.201.109@tcp [ 1157.203313] Key type .llcrypt unregistered [ 1157.204397] Key type ._llcrypt unregistered [ 1166.138864] Key type ._llcrypt registered [ 1166.139852] Key type .llcrypt registered [ 1166.176957] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1166.506533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1166.514600] alg: No test for adler32 (adler32-zlib) [ 1167.375907] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1167.470587] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1169.055203] Key type lgssc registered [ 1169.428339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1172.061398] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1172.069360] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1187.487449] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1192.607309] LustreError: 80543:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 1219.231269] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1221.258229] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1223.601187] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1226.086556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.127585] Lustre: *** cfs_fail_loc=706, val=0*** [ 1226.276050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1226.306903] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1227.678714] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1230.487978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1230.626438] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1231.924442] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1234.264895] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1236.015861] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1239.009583] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.011034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1239.014432] Lustre: Skipped 1 previous similar message [ 1242.765907] Lustre: server umount lustre-OST0000 complete [ 1244.128730] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.129606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1244.133393] Lustre: Skipped 1 previous similar message [ 1244.134952] Lustre: Skipped 1 previous similar message [ 1249.247979] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.249653] Lustre: Skipped 2 previous similar messages [ 1250.072645] Lustre: server umount lustre-MDT0000 complete [ 1251.287200] LustreError: 83201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935554 with bad export cookie 11011356518543535343 [ 1251.290329] LustreError: 83201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1251.290348] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1251.410029] Lustre: server umount lustre-MDT0001 complete [ 1254.799678] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1255.796021] Key type lgssc unregistered [ 1255.916323] LNet: 84156:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1255.919407] LNetError: 84156:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1255.929331] LNet: Removed LNI 192.168.201.109@tcp [ 1256.205441] Key type .llcrypt unregistered [ 1256.206483] Key type ._llcrypt unregistered [ 1265.574587] Key type ._llcrypt registered [ 1265.575555] Key type .llcrypt registered [ 1265.613132] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1266.010898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1266.042821] alg: No test for adler32 (adler32-zlib) [ 1266.888281] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1266.970381] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1268.551129] Key type lgssc registered [ 1268.897739] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1271.410497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1271.415950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1273.751112] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1276.361456] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1276.545044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1277.901420] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1280.796762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1280.905684] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.852270] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1285.593404] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1285.598159] Lustre: MGS: Client 453057bc-36ba-401d-a8f9-a59b59b11688 (at 0@lo) reconnecting [ 1285.605462] LustreError: 86248:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1285.688541] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.715643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1287.005392] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1295.840388] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1295.843296] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1295.848116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1296.865425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1296.866687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1296.867593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1296.875082] Lustre: Skipped 1 previous similar message [ 1300.942520] Lustre: server umount lustre-OST0000 complete [ 1307.104752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1307.104848] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.105550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1307.113467] Lustre: Skipped 1 previous similar message [ 1308.292551] Lustre: server umount lustre-MDT0000 complete [ 1309.657267] Lustre: server umount lustre-MDT0001 complete [ 1310.938726] Lustre: server umount MGS complete [ 1314.645703] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 23:00:16 (1786935616) [ 1323.940135] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 1327.286031] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1339.476788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1340.921414] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1344.730332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1344.814407] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1344.830280] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1344.866249] Lustre: lustre-MDT0001: new disk, initializing [ 1344.894175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1344.902281] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1344.904457] Lustre: 93392:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1344.906859] LustreError: 93392:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1346.368523] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1350.496092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1350.610937] Lustre: lustre-OST0000: new disk, initializing [ 1350.613154] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1350.615984] Lustre: 94159:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1350.636929] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1352.625485] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1356.400297] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1356.768091] Lustre: 94820:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000d44c9755 x1873737752282624/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786935676 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1358.711678] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1358.715898] Lustre: MGS: Client be0ae2fc-f377-4b00-a78e-d8832a5e378b (at 0@lo) reconnecting [ 1358.731259] Lustre: 95216:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1358.740220] LustreError: 93378:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1358.743520] LustreError: 93378:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1358.750763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1358.795842] Lustre: lustre-MDT0000: new disk, initializing [ 1358.842798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1358.851430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1359.903988] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1359.904182] Lustre: 95243:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000008045864a x1873737752288128/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1786935679 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1359.914673] Lustre: 95243:0:(client.c:1639:after_reply()) Skipped 1 previous similar message [ 1360.356215] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1365.858843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 1365.862877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 1368.994763] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 1368.997585] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 1368.997772] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 1389.536330] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1389.540351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1389.545399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1389.548395] Lustre: Skipped 1 previous similar message [ 1395.412406] Lustre: server umount lustre-OST0000 complete [ 1401.312818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1401.312920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1401.313608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1401.313614] Lustre: Skipped 2 previous similar messages [ 1401.321530] Lustre: Skipped 2 previous similar messages [ 1402.895983] Lustre: server umount lustre-MDT0000 complete [ 1410.413026] Lustre: server umount lustre-MDT0001 complete [ 1411.700228] Lustre: server umount MGS complete [ 1415.294141] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 23:01:57 (1786935717) [ 1421.206809] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1422.331237] Key type lgssc unregistered [ 1422.463440] LNet: 97741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1422.465617] LNetError: 97741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1422.473462] LNet: Removed LNI 192.168.201.109@tcp [ 1422.819132] Key type .llcrypt unregistered [ 1422.820088] Key type ._llcrypt unregistered [ 1429.690105] Key type ._llcrypt registered [ 1429.691253] Key type .llcrypt registered [ 1429.728333] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1430.050204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1430.083231] alg: No test for adler32 (adler32-zlib) [ 1430.931922] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1431.016836] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1432.591143] Key type lgssc registered [ 1432.952606] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1435.565712] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1435.572948] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1436.712838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1438.070723] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1440.881643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1440.993548] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1442.312559] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1444.671911] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1446.399444] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1448.764395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1448.880906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1450.941527] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1453.321934] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1462.184571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 1467.362851] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1467.362892] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1467.363630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1467.370901] Lustre: Skipped 1 previous similar message [ 1468.492334] Lustre: server umount lustre-OST0000 complete [ 1471.954524] Lustre: server umount lustre-MDT0000 complete [ 1471.967742] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1471.969887] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1471.975570] LustreError: 98940:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1471.980087] LustreError: 98940:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1472.480407] LustreError: 99668:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1472.484508] LustreError: 99668:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1473.183244] LustreError: 98920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935776 with bad export cookie 6225184420595412250 [ 1473.186877] LustreError: 98920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1473.187128] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1473.311042] Lustre: server umount lustre-MDT0001 complete [ 1477.032622] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1478.043015] Key type lgssc unregistered [ 1478.168383] LNet: 102493:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1478.170888] LNetError: 102493:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1478.183306] LNet: Removed LNI 192.168.201.109@tcp [ 1478.485283] Key type .llcrypt unregistered [ 1478.486837] Key type ._llcrypt unregistered [ 1484.887480] Key type ._llcrypt registered [ 1484.888571] Key type .llcrypt registered [ 1484.930796] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1485.355166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1485.388772] alg: No test for adler32 (adler32-zlib) [ 1486.270094] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1486.378186] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1487.959245] Key type lgssc registered [ 1488.378896] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1490.575639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1490.577252] LustreError: 103554:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1490.577967] Lustre: Skipped 1 previous similar message [ 1490.582635] LustreError: 103553:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1490.585670] LustreError: 103554:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1490.587743] LustreError: 103554:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1490.590926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1490.592033] LustreError: 103554:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1491.763566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1494.811750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1494.913961] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1496.276736] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1497.999615] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lsmod [ 1498.969540] LustreError: 104759:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 1498.969690] LustreError: 104760:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 1498.973750] LustreError: 104759:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4998 [ 1498.975916] LustreError: 104759:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1498.979590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1498.980539] LustreError: 104759:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1499.109704] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1500.127882] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1500.130217] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1500.134364] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.224387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1504.225803] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1504.226649] Lustre: Skipped 1 previous similar message [ 1504.232022] Lustre: Skipped 1 previous similar message [ 1505.250351] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1505.252643] Lustre: Skipped 2 previous similar messages [ 1506.208210] Lustre: server umount lustre-MDT0000 complete [ 1507.564257] LustreError: 103567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935811 with bad export cookie 7590658584611514250 [ 1507.567869] LustreError: 103567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1507.568678] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1509.343242] LustreError: 104022:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000364edf4e x1873737981901824/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1509.395442] Lustre: server umount lustre-MDT0001 complete [ 1520.853047] Lustre: server umount lustre-OST0000 complete [ 1523.693133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1523.866194] LustreError: 105825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1523.872268] LustreError: 105825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1523.894539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1525.227538] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1528.019047] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1528.156907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1529.595391] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1532.103318] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1534.002508] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1536.290717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1536.414560] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1538.365865] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1540.877369] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1548.711118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 1553.890848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1553.891312] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1553.892907] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1553.894263] Lustre: Skipped 1 previous similar message [ 1558.859938] Lustre: server umount lustre-OST0000 complete [ 1564.128564] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1564.128925] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1564.130642] LustreError: Skipped 1 previous similar message [ 1564.134756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1564.137352] Lustre: Skipped 2 previous similar messages [ 1566.296918] Lustre: server umount lustre-MDT0000 complete [ 1567.534200] LustreError: 105805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935871 with bad export cookie 7590658584611514908 [ 1567.537538] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1567.538489] LustreError: 105805:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1567.648908] Lustre: server umount lustre-MDT0001 complete [ 1571.484454] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1572.509649] Key type lgssc unregistered [ 1572.636315] LNet: 109377:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1572.638636] LNetError: 109377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1572.645303] LNet: Removed LNI 192.168.201.109@tcp [ 1572.938991] Key type .llcrypt unregistered [ 1572.940294] Key type ._llcrypt unregistered [ 1582.055581] Key type ._llcrypt registered [ 1582.056692] Key type .llcrypt registered [ 1582.093832] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1582.586360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1582.607196] alg: No test for adler32 (adler32-zlib) [ 1583.467331] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1583.558700] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1585.143124] Key type lgssc registered [ 1585.501773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1588.034383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1588.039759] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1589.187157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1590.590146] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1593.405662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1593.532381] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1594.998783] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1597.370774] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1599.127166] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1601.336972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1601.466370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1603.413335] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1605.830101] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1613.735097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 1618.376807] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1619.194220] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1624.033149] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1624.034781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1624.035367] LustreError: Skipped 1 previous similar message [ 1624.035983] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1624.041225] Lustre: Skipped 1 previous similar message [ 1626.255492] Lustre: server umount lustre-OST0000 complete [ 1629.155088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.156067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1629.161041] Lustre: Skipped 1 previous similar message [ 1629.163864] Lustre: Skipped 1 previous similar message [ 1633.654432] Lustre: server umount lustre-MDT0000 complete [ 1634.272495] LustreError: 110768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.276373] LustreError: 110768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1635.079627] LustreError: 110750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786935938 with bad export cookie 14415324549735369333 [ 1635.083511] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1635.086781] LustreError: 110750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1635.220975] Lustre: server umount lustre-MDT0001 complete [ 1639.281567] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1640.340831] Key type lgssc unregistered [ 1640.472395] LNet: 114424:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1640.475257] LNetError: 114424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1640.481307] LNet: Removed LNI 192.168.201.109@tcp [ 1640.794090] Key type .llcrypt unregistered [ 1640.795097] Key type ._llcrypt unregistered [ 1648.059625] Key type ._llcrypt registered [ 1648.060945] Key type .llcrypt registered [ 1648.097227] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1648.473919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1648.538602] alg: No test for adler32 (adler32-zlib) [ 1649.400451] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1649.500253] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1651.095187] Key type lgssc registered [ 1651.459140] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1654.315188] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1654.321108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1655.459129] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1656.874539] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1659.716351] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1659.847485] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1661.252114] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1663.962679] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1666.075359] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1668.396392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1668.518845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1670.424291] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1673.045627] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1679.783379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 1690.083025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1690.083268] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.084110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1690.085037] LustreError: Skipped 1 previous similar message [ 1690.088297] Lustre: Skipped 1 previous similar message [ 1692.243454] Lustre: server umount lustre-OST0000 complete [ 1695.200900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1695.201711] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1695.205911] Lustre: Skipped 1 previous similar message [ 1695.209229] Lustre: Skipped 2 previous similar messages [ 1699.672949] Lustre: server umount lustre-MDT0000 complete [ 1700.320258] LustreError: 115624:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1700.327539] LustreError: 115624:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1700.938245] LustreError: 115604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936004 with bad export cookie 11050853338214900628 [ 1700.942272] LustreError: 115604:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1700.942448] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1701.065334] Lustre: server umount lustre-MDT0001 complete [ 1704.568288] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1705.598095] Key type lgssc unregistered [ 1705.728447] LNet: 119177:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1705.730982] LNetError: 119177:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1705.742280] LNet: Removed LNI 192.168.201.109@tcp [ 1706.079592] Key type .llcrypt unregistered [ 1706.080726] Key type ._llcrypt unregistered [ 1714.626548] Key type ._llcrypt registered [ 1714.627588] Key type .llcrypt registered [ 1714.662522] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1714.953031] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1714.961054] alg: No test for adler32 (adler32-zlib) [ 1715.812316] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1715.898146] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1717.479153] Key type lgssc registered [ 1717.836213] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1720.282855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1720.288738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1721.439971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1722.715421] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1725.590528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.763092] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1727.417750] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1730.209976] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1732.663428] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1735.314269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1735.446539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.338904] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1739.857275] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1745.702646] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 1752.968983] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1754.147466] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1754.151104] Lustre: Skipped 2 previous similar messages [ 1755.462886] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1755.465110] Lustre: Skipped 2 previous similar messages [ 1759.740851] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1759.751696] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1762.900058] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.9@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1762.903152] Lustre: Skipped 2 previous similar messages [ 1768.953292] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.9@tcp 192.168.0.[2,10]@tcp [ 1773.753975] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.9/24@tcp in log lustre-MDT0000 [ 1773.756501] Lustre: Skipped 2 previous similar messages [ 1776.246210] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.9/24@tcp [ 1776.248371] Lustre: Skipped 1 previous similar message [ 1782.751814] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1782.754348] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1782.758511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1786.850524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1786.850812] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1786.850875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1787.917849] Lustre: server umount lustre-OST0000 complete [ 1791.968954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1791.969541] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1791.972263] Lustre: Skipped 1 previous similar message [ 1791.974122] Lustre: Skipped 1 previous similar message [ 1795.280839] Lustre: server umount lustre-MDT0000 complete [ 1796.465995] LustreError: 120550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936099 with bad export cookie 3794726140267724867 [ 1796.468771] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1796.469330] LustreError: 120550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1796.586836] Lustre: server umount lustre-MDT0001 complete [ 1800.076438] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1801.044059] Key type lgssc unregistered [ 1801.169354] LNet: 125666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1801.172249] LNetError: 125666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1801.180289] LNet: Removed LNI 192.168.201.109@tcp [ 1801.494269] Key type .llcrypt unregistered [ 1801.495513] Key type ._llcrypt unregistered [ 1811.525478] Key type ._llcrypt registered [ 1811.526541] Key type .llcrypt registered [ 1811.565491] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1811.921777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1811.970692] alg: No test for adler32 (adler32-zlib) [ 1812.857366] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1812.961933] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1814.543176] Key type lgssc registered [ 1814.933932] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1817.467409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1817.474965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1818.619804] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 1818.621535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1818.627776] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.9/24@tcp [ 1819.917916] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1822.893686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.014710] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1823.023448] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.9/24@tcp [ 1823.044350] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1824.393785] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1826.806184] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1828.653397] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1830.906271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1831.020312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1832.954921] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1835.362342] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1844.328378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 1848.801342] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1848.804817] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1848.809474] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.828326] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1849.828771] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1849.837918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1850.829696] Lustre: server umount lustre-OST0000 complete [ 1853.919878] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1853.923640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1853.929091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1853.930808] Lustre: Skipped 1 previous similar message [ 1854.165651] Lustre: server umount lustre-MDT0000 complete [ 1854.944387] LustreError: 127948:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1854.949610] LustreError: 127948:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1855.381077] LustreError: 127182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936158 with bad export cookie 18082622161857415029 [ 1855.383444] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1855.384376] LustreError: 127182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1855.494730] Lustre: server umount lustre-MDT0001 complete [ 1859.083872] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1860.058729] Key type lgssc unregistered [ 1860.189388] LNet: 130808:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1860.193335] LNetError: 130808:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1860.201302] LNet: Removed LNI 192.168.201.109@tcp [ 1860.498677] Key type .llcrypt unregistered [ 1860.499699] Key type ._llcrypt unregistered [ 1868.829933] Key type ._llcrypt registered [ 1868.830978] Key type .llcrypt registered [ 1868.867881] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 1875.045781] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1875.434736] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1875.459072] alg: No test for adler32 (adler32-zlib) [ 1876.324131] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 1876.436228] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1878.023114] Key type lgssc registered [ 1878.406359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1889.451194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1889.455990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1890.548250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1890.564646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1890.601204] Lustre: lustre-MDT0000: new disk, initializing [ 1890.632417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1890.641613] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1892.261194] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1896.827522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1896.868498] Lustre: 134177:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1896.889957] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1896.891569] Lustre: Skipped 1 previous similar message [ 1896.926995] Lustre: lustre-MDT0001: new disk, initializing [ 1896.958207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1896.969040] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1896.974422] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1898.248474] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1901.873837] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1903.691410] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1905.979207] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1906.091177] Lustre: lustre-OST0000: new disk, initializing [ 1906.092921] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1906.095426] Lustre: 136146:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1906.128831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1908.013486] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1911.553357] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1912.916916] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1914.869570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1914.872436] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1914.901200] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1915.014866] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1916.304111] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1916.374548] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1919.971019] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1919.971358] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1919.972129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1919.979640] Lustre: Skipped 1 previous similar message [ 1923.090362] Lustre: server umount lustre-OST0000 complete [ 1925.088591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1925.089373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1925.091919] Lustre: Skipped 1 previous similar message [ 1925.095077] Lustre: Skipped 2 previous similar messages [ 1930.208229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1930.209986] Lustre: Skipped 1 previous similar message [ 1930.462537] Lustre: server umount lustre-MDT0000 complete [ 1931.708173] LustreError: 134171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936235 with bad export cookie 5063299517262562525 [ 1931.711648] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1931.711768] LustreError: 134171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1931.824929] Lustre: server umount lustre-MDT0001 complete [ 1934.564893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1934.737704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1936.036533] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1939.154994] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1939.275209] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1940.696086] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1943.102216] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1944.884948] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1947.171448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1947.280450] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.189863] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1951.697245] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1953.072524] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1961.340084] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 1962.878136] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1962.961990] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1966.604095] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1966.668684] Lustre: lustre-OST0001: new disk, initializing [ 1966.670585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 1966.673152] Lustre: 141798:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1966.698688] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.693547] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1972.170372] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 1973.244219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 1973.248137] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 1973.279961] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 1974.274424] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 1975.688556] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 1975.792936] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1977.753128] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90ecc88e6000.ost_server_uuid 50 [ 1978.247260] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90ecc88e6000.ost_server_uuid in FULL state after 0 sec [ 1983.464253] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1983.465093] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1983.468578] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 1983.469814] LustreError: Skipped 2 previous similar messages [ 1983.471371] Lustre: Skipped 1 previous similar message [ 1986.246665] Lustre: server umount lustre-OST0001 complete [ 1988.577865] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1988.578568] LustreError: 140197:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.580298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1988.581248] Lustre: Skipped 2 previous similar messages [ 1988.585291] LustreError: 140197:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1993.618463] Lustre: server umount lustre-OST0000 complete [ 1995.743865] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1995.746183] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1998.816592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1998.818142] Lustre: Skipped 3 previous similar messages [ 2000.993045] Lustre: server umount lustre-MDT0000 complete [ 2002.441247] LustreError: 143115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936305 with bad export cookie 5063299517262563393 [ 2002.448606] LustreError: 143115:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2002.450552] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2002.626505] Lustre: server umount lustre-MDT0001 complete [ 2009.306168] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2010.349941] Key type lgssc unregistered [ 2010.474438] LNet: 144816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2010.478479] LNetError: 144816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2010.489331] LNet: Removed LNI 192.168.201.109@tcp [ 2010.802058] Key type .llcrypt unregistered [ 2010.803461] Key type ._llcrypt unregistered [ 2023.606559] Key type ._llcrypt registered [ 2023.607556] Key type .llcrypt registered [ 2023.642945] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2029.586521] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2030.005389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2030.016344] alg: No test for adler32 (adler32-zlib) [ 2030.862358] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2030.944672] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2032.527181] Key type lgssc registered [ 2032.894438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2044.786537] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2044.793255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2045.876983] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2045.886481] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2045.926047] Lustre: lustre-MDT0000: new disk, initializing [ 2045.945309] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2045.950811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2047.274937] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2051.628491] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.670811] Lustre: 148940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2051.694933] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2051.696794] Lustre: Skipped 1 previous similar message [ 2051.737083] Lustre: lustre-MDT0001: new disk, initializing [ 2051.768707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2051.782258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2051.785497] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2053.113483] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2057.506960] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2059.468374] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2061.755782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2061.858814] Lustre: lustre-OST0000: new disk, initializing [ 2061.860309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2061.862640] Lustre: 150908:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2061.889435] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2063.857845] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2067.573960] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2068.982788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2068.987423] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2069.014423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2074.912546] Lustre: Failing over lustre-OST0000 [ 2074.976806] Lustre: server umount lustre-OST0000 complete [ 2087.969097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2088.113353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.119899] Lustre: Skipped 1 previous similar message [ 2088.125590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2088.130646] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2089.499665] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2090.091243] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2091.365665] Lustre: Failing over lustre-MDT0000 [ 2091.462035] Lustre: server umount lustre-MDT0000 complete [ 2093.023913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2093.026518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.559265] Lustre: 147169:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786936382/real 1786936382] req@0000000015d753ab x1873738553388928/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1786936398 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2098.655993] LustreError: 148950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2098.660647] LustreError: 148950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2099.679144] Lustre: 147171:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786936387/real 1786936387] req@000000000f15524d x1873738553390848/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1786936403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2099.729609] LustreError: 148945:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.776064] LustreError: 150717:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2103.780318] LustreError: 150717:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2104.321151] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.361526] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2104.365234] Lustre: 148943:0:(mgc_request.c:1899:mgc_process_log()) MGC192.168.201.109@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2104.433559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2104.451641] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2104.478472] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2104.850044] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2105.770168] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2109.927419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2109.934752] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 2109.938224] Lustre: Skipped 1 previous similar message [ 2109.955302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2115.041109] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2115.042653] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2115.044562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2115.047642] Lustre: Skipped 3 previous similar messages [ 2115.050533] Lustre: Skipped 1 previous similar message [ 2117.070180] Lustre: server umount lustre-OST0000 complete [ 2120.161029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2120.161394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2120.161853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2120.164831] Lustre: Skipped 1 previous similar message [ 2120.166776] LustreError: Skipped 1 previous similar message [ 2124.377067] Lustre: server umount lustre-MDT0000 complete [ 2125.279944] LustreError: 148950:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2125.284406] LustreError: 148950:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2125.594200] LustreError: 148931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936429 with bad export cookie 4922005000578197199 [ 2125.597092] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2125.597370] LustreError: 148931:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2125.710721] Lustre: server umount lustre-MDT0001 complete [ 2129.506408] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2130.540535] Key type lgssc unregistered [ 2130.658345] LNet: 154557:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2130.660995] LNetError: 154557:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2130.667325] LNet: Removed LNI 192.168.201.109@tcp [ 2130.972758] Key type .llcrypt unregistered [ 2130.974034] Key type ._llcrypt unregistered [ 2150.771144] Key type ._llcrypt registered [ 2150.772644] Key type .llcrypt registered [ 2150.813240] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2151.278616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2151.299369] alg: No test for adler32 (adler32-zlib) [ 2152.171225] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2152.264156] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2153.839121] Key type lgssc registered [ 2154.165306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2156.561764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2156.567283] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.703562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.013434] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2162.646543] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2164.834993] LustreError: 156102:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2164.840769] LustreError: 156102:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2164.844879] LustreError: 156102:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2164.975171] Lustre: server umount lustre-MDT0000 complete [ 2169.888710] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2173.324743] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2185.021515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2185.102805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2185.112418] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2185.151181] Lustre: lustre-MDT0000: new disk, initializing [ 2185.172189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.178049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2186.509251] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2190.655613] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2190.700447] Lustre: 159402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2190.729721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2190.732776] Lustre: Skipped 1 previous similar message [ 2190.775257] Lustre: lustre-MDT0001: new disk, initializing [ 2190.812484] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2190.824586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2190.828188] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2192.201070] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2196.018757] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2197.929269] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2200.312436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2200.414103] Lustre: lustre-OST0000: new disk, initializing [ 2200.416691] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2200.419847] Lustre: 161367:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2200.450907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2202.348923] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2205.976470] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2209.277210] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2209.281822] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2209.319810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2212.981438] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2213.514505] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 23:15:15 (1786936515) [ 2275.807594] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2275.807971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2275.810919] Lustre: Skipped 1 previous similar message [ 2275.813450] Lustre: Skipped 2 previous similar messages [ 2280.927599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2280.930262] Lustre: Skipped 1 previous similar message [ 2282.086638] Lustre: server umount lustre-MDT0000 complete [ 2283.296596] LustreError: 159393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936586 with bad export cookie 8109784736675926832 [ 2283.297424] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2283.300312] LustreError: 159393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2286.047432] LustreError: 159407:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2286.047488] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2286.054060] LustreError: 159407:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2286.058219] Lustre: Skipped 1 previous similar message [ 2286.058505] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2288.095399] LustreError: 159408:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2288.095433] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2289.442723] Lustre: server umount lustre-MDT0001 complete [ 2304.991157] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2305.042068] Lustre: server umount lustre-OST0000 complete [ 2308.139816] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2311.347835] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2322.893676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2322.974908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2322.984860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2323.021262] Lustre: lustre-MDT0000: new disk, initializing [ 2323.041943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2323.048303] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2324.408527] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2328.413812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2328.456456] Lustre: 165993:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2328.481761] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2328.484356] Lustre: Skipped 1 previous similar message [ 2328.524839] Lustre: lustre-MDT0001: new disk, initializing [ 2328.568716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2328.574116] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2329.895558] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2333.636814] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2335.484763] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2337.707828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2337.805229] Lustre: lustre-OST0000: new disk, initializing [ 2337.806928] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2337.810871] Lustre: 167957:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2337.840977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2337.843899] Lustre: Skipped 1 previous similar message [ 2339.712619] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2340.023867] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2340.026744] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2340.061994] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2343.359258] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2344.797409] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2344.867931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2346.234949] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2346.305478] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2350.561257] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2350.562315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2350.563581] LustreError: Skipped 1 previous similar message [ 2350.564054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2350.567235] Lustre: Skipped 1 previous similar message [ 2360.801767] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2360.804313] Lustre: Skipped 4 previous similar messages [ 2361.311132] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2361.356462] Lustre: server umount lustre-OST0000 complete [ 2364.383747] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2364.385840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2368.791306] Lustre: server umount lustre-MDT0000 complete [ 2370.060330] LustreError: 165984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936673 with bad export cookie 8109784736676085648 [ 2370.063319] LustreError: 165984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2370.063880] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2370.201139] Lustre: server umount lustre-MDT0001 complete [ 2373.086928] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:17:55 (1786936675) [ 2378.231826] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2381.455349] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2393.161321] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2393.244142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2393.254745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2393.298376] Lustre: lustre-MDT0000: new disk, initializing [ 2393.321183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.327346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2394.675652] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2398.675089] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2398.718732] Lustre: 172706:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2398.816704] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2400.118908] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2403.730595] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2405.522048] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2407.847369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2407.971993] Lustre: 174684:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2410.046258] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2413.800504] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2415.608805] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2415.646044] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2425.826604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2425.826986] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2425.829349] LustreError: Skipped 1 previous similar message [ 2425.832523] Lustre: Skipped 1 previous similar message [ 2425.833533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2425.836374] Lustre: Skipped 4 previous similar messages [ 2427.083340] Lustre: server umount lustre-OST0000 complete [ 2429.407862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2434.457612] Lustre: server umount lustre-MDT0000 complete [ 2435.732691] LustreError: 173506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936739 with bad export cookie 8109784736676086516 [ 2435.737267] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2435.869926] Lustre: server umount lustre-MDT0001 complete [ 2438.713674] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 23:19:01 (1786936741) [ 2443.826971] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2446.999640] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2458.433362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.511582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2458.520459] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2458.522219] Lustre: Skipped 3 previous similar messages [ 2458.557285] Lustre: lustre-MDT0000: new disk, initializing [ 2458.558729] Lustre: Skipped 2 previous similar messages [ 2458.580367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2458.582365] Lustre: Skipped 2 previous similar messages [ 2458.587525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2458.589915] Lustre: Skipped 2 previous similar messages [ 2459.878579] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2463.858566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2463.902806] Lustre: 178996:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2463.999447] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2465.282906] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2469.091860] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2470.999924] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2473.452366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2473.561866] Lustre: 180972:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2475.652452] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2479.502060] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2482.214619] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2487.271460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.271838] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2487.273341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2487.273349] Lustre: Skipped 3 previous similar messages [ 2487.280355] Lustre: Skipped 4 previous similar messages [ 2487.285056] LustreError: Skipped 1 previous similar message [ 2489.193385] Lustre: server umount lustre-OST0000 complete [ 2492.785532] Lustre: server umount lustre-MDT0000 complete [ 2494.689905] LustreError: 179034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936798 with bad export cookie 8109784736676087587 [ 2494.697336] LustreError: 179034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2494.697643] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2494.858428] Lustre: server umount lustre-MDT0001 complete [ 2498.778895] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2499.679800] Key type lgssc unregistered [ 2499.795325] LNet: 182827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2499.797386] LNetError: 182827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2499.803338] LNet: Removed LNI 192.168.201.109@tcp [ 2500.095547] Key type .llcrypt unregistered [ 2500.096532] Key type ._llcrypt unregistered [ 2508.874961] Key type ._llcrypt registered [ 2508.876074] Key type .llcrypt registered [ 2508.908509] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2509.213305] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2509.259690] alg: No test for adler32 (adler32-zlib) [ 2510.114026] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2510.205895] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2511.783220] Key type lgssc registered [ 2512.154221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2514.641525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2514.647458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2515.788147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2517.142545] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2520.007463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2520.137494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2521.606201] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2524.026532] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2525.849048] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2528.043069] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2528.147099] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.996288] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2532.318824] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2541.478370] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2555.872627] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2555.875071] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2555.879087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2556.898203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2556.899101] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2556.899182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2556.900249] Lustre: Skipped 1 previous similar message [ 2560.714942] Lustre: server umount lustre-OST0000 complete [ 2562.016776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2562.017336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2562.021981] Lustre: Skipped 1 previous similar message [ 2562.025635] Lustre: Skipped 1 previous similar message [ 2567.136391] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2567.139195] Lustre: Skipped 1 previous similar message [ 2572.256173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2572.257957] Lustre: Skipped 1 previous similar message [ 2576.351075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2576.408443] Lustre: server umount lustre-MDT0000 complete [ 2577.376473] LustreError: 185653:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2577.380757] LustreError: 185653:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2577.620272] LustreError: 184954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936881 with bad export cookie 11499499761254841019 [ 2577.624225] LustreError: 184954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2577.625470] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2577.749331] Lustre: server umount lustre-MDT0001 complete [ 2581.177803] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2582.232631] Key type lgssc unregistered [ 2582.363347] LNet: 187779:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2582.365646] LNetError: 187779:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2582.372319] LNet: Removed LNI 192.168.201.109@tcp [ 2582.693919] Key type .llcrypt unregistered [ 2582.695406] Key type ._llcrypt unregistered [ 2591.168888] Key type ._llcrypt registered [ 2591.169939] Key type .llcrypt registered [ 2591.204641] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2591.581360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2591.593458] alg: No test for adler32 (adler32-zlib) [ 2592.438487] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 2592.527729] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2594.111167] Key type lgssc registered [ 2594.461651] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2596.880917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2596.886529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2598.028199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2599.295794] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2601.989083] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2602.108313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2603.460728] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2605.789787] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2607.677268] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2609.857807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2609.967952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.774192] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2614.122676] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2624.294095] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 2627.551831] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2627.554273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2627.558448] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.603568] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2629.604890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2629.607569] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2629.610109] Lustre: Skipped 1 previous similar message [ 2634.722971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2634.725317] Lustre: Skipped 2 previous similar messages [ 2639.842328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2639.846279] Lustre: Skipped 2 previous similar messages [ 2640.863239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2640.915406] Lustre: server umount lustre-OST0000 complete [ 2643.456124] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff90ecc8a28800.ost_server_uuid 50 [ 2643.959936] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff90ecc8a28800.ost_server_uuid in DISCONN state after 0 sec [ 2644.507969] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 2655.201937] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.202819] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2655.209274] Lustre: Skipped 1 previous similar message [ 2655.214802] Lustre: Skipped 2 previous similar messages [ 2660.648679] Lustre: server umount lustre-MDT0000 complete [ 2662.078699] LustreError: 189152:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786936965 with bad export cookie 11647674305682182542 [ 2662.084729] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.220279] Lustre: server umount lustre-MDT0001 complete [ 2665.083805] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 23:22:47 (1786936967) [ 2667.197829] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2667.364648] LustreError: 193134:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2667.370289] LustreError: 193134:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2667.386458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.620089] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2671.229154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2672.607513] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2674.787038] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2676.510057] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2678.676588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2678.787937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2678.790086] Lustre: Skipped 1 previous similar message [ 2680.606772] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2682.893281] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2685.055574] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2685.139796] Lustre: lustre-OST0001: new disk, initializing [ 2685.141497] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2685.143952] Lustre: 195975:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2686.967794] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2690.561468] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2692.603197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2692.606057] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2692.637075] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2693.673068] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 2698.724150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2698.725224] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2698.725458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2698.725463] Lustre: Skipped 2 previous similar messages [ 2698.728092] Lustre: Skipped 1 previous similar message [ 2698.938263] Lustre: server umount lustre-OST0000 complete [ 2700.903425] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2700.984403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2701.337265] LustreError: 194954:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2702.359536] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2702.426210] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 2703.840415] LustreError: 196695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2703.846545] LustreError: 196695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2706.451499] LustreError: 194955:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.572265] LustreError: 195354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2711.577296] LustreError: 195354:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2713.056635] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2713.060317] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.550501] Lustre: server umount lustre-OST0001 complete [ 2722.783860] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2722.786274] LustreError: Skipped 1 previous similar message [ 2722.787479] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.790849] Lustre: Skipped 1 previous similar message [ 2722.793438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2722.795204] Lustre: Skipped 5 previous similar messages [ 2734.047123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2734.113142] Lustre: server umount lustre-MDT0000 complete [ 2734.560028] LustreError: 193135:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2734.566509] LustreError: 193135:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2735.288245] LustreError: 193116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937038 with bad export cookie 11647674305682183599 [ 2735.291505] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.291656] LustreError: 193116:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 2735.405412] Lustre: server umount lustre-MDT0001 complete [ 2742.881456] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 23:24:05 (1786937045) [ 2745.039057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2745.154556] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2745.165775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2745.243648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2745.246829] Lustre: Skipped 1 previous similar message [ 2746.645261] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2749.358038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.396872] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2749.410978] Lustre: 199388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2750.869423] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2753.279519] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2755.008377] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2757.176832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.246395] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2759.133684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2761.398885] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2763.293784] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 2763.622343] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2763.647107] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2763.721677] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2763.723697] Lustre: Skipped 2 previous similar messages [ 2765.599677] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2768.043620] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2770.417116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2770.419475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2770.420408] Lustre: Skipped 2 previous similar messages [ 2770.422625] Lustre: Skipped 4 previous similar messages [ 2770.447425] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 2783.199085] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2783.233484] Lustre: server umount lustre-OST0000 complete [ 2783.763318] LustreError: 201237:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.003514] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2796.007019] LustreError: Skipped 1 previous similar message [ 2799.957652] Lustre: server umount lustre-OST0001 complete [ 2805.728034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2805.731347] Lustre: Skipped 2 previous similar messages [ 2807.591933] Lustre: server umount lustre-MDT0000 complete [ 2809.035605] LustreError: 199379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937112 with bad export cookie 11647674305682184985 [ 2809.039950] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2809.180786] Lustre: server umount lustre-MDT0001 complete [ 2817.805654] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 23:25:19 (1786937119) [ 2824.053252] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2828.497775] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2839.704860] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.787762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2839.798475] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2839.837753] Lustre: lustre-MDT0000: new disk, initializing [ 2839.859698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2839.866380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2841.664372] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2846.953118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2847.028670] Lustre: 207450:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2847.071859] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2847.075564] Lustre: Skipped 1 previous similar message [ 2847.123682] Lustre: lustre-MDT0001: new disk, initializing [ 2847.166545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2847.170790] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2848.780902] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2852.787394] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2854.679711] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2856.886309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2856.976815] Lustre: lustre-OST0000: new disk, initializing [ 2856.978244] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2856.980425] Lustre: 209417:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2858.745347] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2859.253846] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2859.256738] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2859.285406] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2862.239033] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2863.540823] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2863.601897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2864.853344] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2864.926987] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2867.598280] Lustre: server umount lustre-OST0000 complete [ 2869.729627] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2869.730714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2869.732939] Lustre: Skipped 2 previous similar messages [ 2869.735410] Lustre: Skipped 13 previous similar messages [ 2878.968642] Lustre: server umount lustre-MDT0000 complete [ 2879.968053] LustreError: 208982:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2879.972439] LustreError: 208982:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 2880.138035] LustreError: 208244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937183 with bad export cookie 11647674305682186728 [ 2880.141407] LustreError: 208244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2880.141651] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2889.743159] Lustre: server umount lustre-MDT0001 complete [ 2892.532985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2893.906925] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2896.619612] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2898.017287] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2900.260693] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2902.068364] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2904.215770] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2904.324943] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2904.328109] Lustre: Skipped 4 previous similar messages [ 2906.117505] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2908.375977] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2909.653142] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2917.852838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2919.039975] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2919.102163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2922.977381] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2922.980810] LustreError: Skipped 1 previous similar message [ 2925.768799] Lustre: server umount lustre-OST0000 complete [ 2927.709470] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2927.790755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 2929.180371] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2952.617549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2954.512426] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2956.729327] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2957.193501] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2960.206119] Lustre: server umount lustre-OST0000 complete [ 2963.551129] Lustre: server umount lustre-MDT0000 complete [ 2963.936859] LustreError: 212371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2963.941843] LustreError: 212371:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2964.810315] LustreError: 211610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937268 with bad export cookie 11647674305682187610 [ 2964.814589] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2964.816307] LustreError: 211610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2964.940180] Lustre: server umount lustre-MDT0001 complete [ 2967.767498] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 23:27:50 (1786937270) [ 2969.882990] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2971.399834] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2974.097198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2975.480672] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2977.829412] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2979.695214] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2981.957497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2983.966171] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2986.281282] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2987.841731] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2996.107433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 8 sec [ 2997.495189] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2997.572736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2999.694546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2999.769937] Lustre: lustre-OST0001: new disk, initializing [ 2999.771473] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2999.773612] Lustre: 221085:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3001.704911] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3005.390912] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3006.735794] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3009.024379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3009.028490] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3009.054325] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3009.857246] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 3011.182208] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3011.246835] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3014.116106] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3014.116605] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.119159] LustreError: Skipped 2 previous similar messages [ 3014.122224] Lustre: Skipped 4 previous similar messages [ 3014.122688] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3014.125902] Lustre: Skipped 7 previous similar messages [ 3017.917713] Lustre: server umount lustre-OST0001 complete [ 3019.721203] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3019.790274] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3021.081613] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3021.146676] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3045.461865] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3045.534480] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3045.536664] Lustre: Skipped 5 previous similar messages [ 3047.590729] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3050.359557] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3050.836096] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3050.984320] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3050.988684] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3050.989049] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3050.993973] Lustre: Skipped 1 previous similar message [ 3053.509395] Lustre: server umount lustre-OST0001 complete [ 3071.506570] Lustre: server umount lustre-OST0000 complete [ 3087.327084] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3087.391819] Lustre: server umount lustre-MDT0000 complete [ 3088.552167] LustreError: 217643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937392 with bad export cookie 11647674305682188541 [ 3088.555662] LustreError: 217643:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3088.555780] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3094.768188] Lustre: server umount lustre-MDT0001 complete [ 3102.300645] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 23:30:04 (1786937404) [ 3104.396803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.493213] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3104.504533] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3105.790554] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3108.358379] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3108.388653] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3108.397104] Lustre: 226340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3109.734951] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3111.983291] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3113.665813] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3115.884388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.951091] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3117.766687] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3119.926206] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3123.413333] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3125.274789] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3127.542848] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3128.032932] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3129.648857] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3129.732157] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3131.288584] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3131.352226] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3133.814451] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff90ecc8ab7800.ost_server_uuid 50 [ 3134.482277] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff90ecc8ab7800.ost_server_uuid in FULL state after 0 sec [ 3134.900857] Lustre: Permanently deactivating lustre-OST0001 [ 3134.906642] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3135.789673] Lustre: Permanently reactivating lustre-OST0001 [ 3141.600722] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3141.603067] LustreError: Skipped 4 previous similar messages [ 3142.908704] Lustre: server umount lustre-OST0001 complete [ 3146.723878] LustreError: 228183:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.729103] LustreError: 228183:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 27 previous similar messages [ 3150.556016] Lustre: server umount lustre-OST0000 complete [ 3158.516313] Lustre: server umount lustre-MDT0000 complete [ 3160.170141] LustreError: 226332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937463 with bad export cookie 11647674305682189815 [ 3160.172806] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3160.174942] LustreError: 226332:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3160.286651] Lustre: server umount lustre-MDT0001 complete [ 3169.910377] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 23:31:12 (1786937472) [ 3172.308038] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.401870] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3172.410587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3172.413281] Lustre: Skipped 5 previous similar messages [ 3173.831986] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3176.783216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3176.817836] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3176.819328] Lustre: Skipped 1 previous similar message [ 3176.828252] Lustre: 233015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3178.268148] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3180.936243] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3182.885540] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3185.164697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.238672] Lustre: Permanently deactivating lustre-OST0000 [ 3185.243141] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3186.544064] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3187.168480] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3189.609419] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3192.363567] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3193.643841] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3194.490888] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3197.161508] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3197.660438] Lustre: Permanently reactivating lustre-OST0000 [ 3200.435270] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3200.456401] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3213.637759] Lustre: server umount lustre-OST0001 complete [ 3221.265452] Lustre: server umount lustre-OST0000 complete [ 3228.775552] Lustre: server umount lustre-MDT0000 complete [ 3230.173578] LustreError: 236603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937533 with bad export cookie 11647674305682191453 [ 3230.178836] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3230.323674] Lustre: server umount lustre-MDT0001 complete [ 3234.305414] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3235.287695] Key type lgssc unregistered [ 3235.405452] LNet: 237868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3235.409268] LNetError: 237868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3235.419396] LNet: Removed LNI 192.168.201.109@tcp [ 3235.725717] Key type .llcrypt unregistered [ 3235.726867] Key type ._llcrypt unregistered [ 3244.079068] Key type ._llcrypt registered [ 3244.080038] Key type .llcrypt registered [ 3244.113064] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3244.435711] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3244.475723] alg: No test for adler32 (adler32-zlib) [ 3245.325120] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3245.416663] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3246.999120] Key type lgssc registered [ 3247.340894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3250.377232] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3250.383353] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3251.519840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3252.756495] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3255.335900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3255.371148] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3255.379975] Lustre: 239301:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3255.383412] Lustre: Permanently deactivating lustre-MDT0001 [ 3255.388026] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3255.484925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3256.728147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3259.659752] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3260.404815] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3260.410748] LustreError: 239342:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3260.419449] LustreError: 239342:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3260.425790] LustreError: 239342:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 8, retries 1, failed: rc = -108 [ 3262.126288] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3264.736292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3264.863061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3267.234684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3270.214480] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3272.604739] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3272.699766] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3274.986292] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3276.781364] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3276.783666] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3277.897782] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3279.648810] Lustre: Permanently reactivating lustre-MDT0001 [ 3279.658201] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3279.662243] Lustre: Skipped 2 previous similar messages [ 3279.854466] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3279.854656] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3290.227398] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3290.876147] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3292.719550] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3292.822688] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3294.335726] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3294.400589] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3295.339916] Lustre: Permanently deactivating lustre-MDT0001 [ 3295.349838] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3295.355661] Lustre: Skipped 2 previous similar messages [ 3305.491817] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3310.564659] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3310.566975] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3310.570628] Lustre: Skipped 1 previous similar message [ 3315.104878] Lustre: server umount lustre-MDT0000 complete [ 3315.685574] LustreError: 240723:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.692438] LustreError: 240723:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [ 3316.629705] LustreError: 239291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937620 with bad export cookie 5935907201447995479 [ 3316.632935] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3316.636644] LustreError: 239291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3316.761787] Lustre: server umount lustre-MDT0001 complete [ 3324.357807] Lustre: server umount lustre-OST0000 complete [ 3329.519218] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:33:51 (1786937631) [ 3334.504130] Lustre: server umount lustre-OST0001 complete [ 3337.177770] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3341.560121] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3357.269407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3357.393647] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3357.399449] Lustre: Skipped 2 previous similar messages [ 3357.410901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3357.456496] Lustre: lustre-MDT0000: new disk, initializing [ 3357.479601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3357.486868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3359.168118] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3364.587534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3364.649466] Lustre: 247615:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3364.687814] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3364.691744] Lustre: Skipped 1 previous similar message [ 3364.741560] Lustre: lustre-MDT0001: new disk, initializing [ 3364.794635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3364.799248] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3366.168300] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3370.760436] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3373.458761] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3376.650962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3376.798312] Lustre: lustre-OST0000: new disk, initializing [ 3376.800501] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3376.803928] Lustre: 249581:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3376.845618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3376.849296] Lustre: Skipped 1 previous similar message [ 3379.241140] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3380.738805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3380.743400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3380.780103] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3383.770143] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3384.428693] LustreError: 247622:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3388.994801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3389.062409] Lustre: lustre-OST0001: new disk, initializing [ 3389.064042] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3389.066174] Lustre: 250695:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3391.704687] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3394.519044] LustreError: 247622:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3394.525611] LustreError: 247621:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 3396.372602] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3397.097500] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3398.178709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3398.184645] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3398.249119] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3404.615067] LustreError: 247621:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 3405.864232] Lustre: lustre-OST0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 3405.867861] Lustre: Skipped 3 previous similar messages [ 3408.359740] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3408.364322] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3408.367082] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3408.371726] Lustre: Skipped 3 previous similar messages [ 3410.967298] Lustre: lustre-OST0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 3410.971904] Lustre: Skipped 1 previous similar message [ 3411.350162] Lustre: server umount lustre-OST0001 complete [ 3413.482110] LustreError: 249565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3413.491861] LustreError: 249565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3418.597044] LustreError: 249566:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.606311] LustreError: 249566:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3423.720573] LustreError: 249565:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.730455] LustreError: 249565:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3426.785995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3426.786112] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3426.790840] LustreError: Skipped 1 previous similar message [ 3426.799493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3426.804631] Lustre: Skipped 1 previous similar message [ 3428.840395] LustreError: 251776:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3428.845652] LustreError: 251776:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3430.458863] Lustre: server umount lustre-OST0000 complete [ 3433.955847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3433.963840] Lustre: Skipped 2 previous similar messages [ 3438.509170] Lustre: server umount lustre-MDT0000 complete [ 3439.072761] LustreError: 248425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.082729] LustreError: 248425:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3440.075234] LustreError: 247608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937743 with bad export cookie 5935907201447998286 [ 3440.082454] LustreError: 247608:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3440.082710] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.275674] Lustre: server umount lustre-MDT0001 complete [ 3450.937400] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3452.449306] Key type lgssc unregistered [ 3452.640833] LNet: 252731:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3452.644817] LNetError: 252731:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3452.656666] LNet: Removed LNI 192.168.201.109@tcp [ 3453.161804] Key type .llcrypt unregistered [ 3453.164086] Key type ._llcrypt unregistered [ 3470.517198] Key type ._llcrypt registered [ 3470.518969] Key type .llcrypt registered [ 3470.575793] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3470.930655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3470.972521] alg: No test for adler32 (adler32-zlib) [ 3471.810737] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3471.890053] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3473.463162] Key type lgssc registered [ 3474.039299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3477.315570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3477.323800] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3478.415730] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3478.421652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3478.471364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3479.664994] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3482.246298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3482.275464] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3482.284684] Lustre: 254916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3482.369429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3483.568248] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3485.709676] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3487.363948] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3489.325680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3489.388392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3489.443595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3491.159954] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3493.280473] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3499.529951] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 3545.569603] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3545.570442] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3545.577462] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3545.579444] Lustre: Skipped 1 previous similar message [ 3548.325624] Lustre: server umount lustre-OST0000 complete [ 3549.761680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 3555.052778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3555.193552] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3557.298455] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3559.865736] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3566.437264] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3566.441906] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3566.442074] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3566.446284] Lustre: Skipped 1 previous similar message [ 3597.284242] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.285664] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3597.286309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3597.288255] Lustre: Skipped 2 previous similar messages [ 3597.290246] LustreError: Skipped 2 previous similar messages [ 3598.880548] Lustre: server umount lustre-OST0000 complete [ 3600.353091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3600.357059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.363806] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3600.366929] Lustre: Skipped 1 previous similar message [ 3602.400597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.401079] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.485634] Lustre: server umount lustre-MDT0000 complete [ 3607.809910] LustreError: 255654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937911 with bad export cookie 5640903192101821086 [ 3607.814189] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3607.814897] LustreError: 255654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3607.941287] Lustre: server umount lustre-MDT0001 complete [ 3612.058104] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3612.997140] Key type lgssc unregistered [ 3613.110443] LNet: 260349:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3613.113542] LNetError: 260349:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3613.122335] LNet: Removed LNI 192.168.201.109@tcp [ 3613.408922] Key type .llcrypt unregistered [ 3613.410041] Key type ._llcrypt unregistered [ 3623.441389] Key type ._llcrypt registered [ 3623.443138] Key type .llcrypt registered [ 3623.493979] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3623.979680] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3623.986813] alg: No test for adler32 (adler32-zlib) [ 3624.875940] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3624.994504] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3626.583121] Key type lgssc registered [ 3626.944463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3630.496409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3630.504682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3631.654612] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.353679] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3637.389338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3637.505541] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3638.982323] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3641.901270] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3643.986567] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3646.827301] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3646.945742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3648.786823] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3651.668044] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3657.193873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 3672.547945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3672.549770] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3672.550021] LustreError: Skipped 1 previous similar message [ 3672.553630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3672.555680] Lustre: Skipped 1 previous similar message [ 3675.950270] Lustre: server umount lustre-OST0000 complete [ 3678.687838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3678.690397] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3678.694309] Lustre: Skipped 1 previous similar message [ 3678.695909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3682.785407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3682.787225] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3683.831456] Lustre: server umount lustre-MDT0000 complete [ 3685.543528] LustreError: 263590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786937989 with bad export cookie 14569763114533501352 [ 3685.546431] LustreError: 263590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3685.548265] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3685.700534] Lustre: server umount lustre-MDT0001 complete [ 3690.416829] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3691.692376] Key type lgssc unregistered [ 3691.857443] LNet: 266075:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3691.859879] LNetError: 266075:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3691.867444] LNet: Removed LNI 192.168.201.109@tcp [ 3692.272901] Key type .llcrypt unregistered [ 3692.274681] Key type ._llcrypt unregistered [ 3700.590553] Key type ._llcrypt registered [ 3700.591621] Key type .llcrypt registered [ 3700.623746] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3700.948931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3700.973650] alg: No test for adler32 (adler32-zlib) [ 3701.847746] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3701.947638] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3703.535122] Key type lgssc registered [ 3703.839329] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3706.669483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3706.674925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3707.839613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3709.501726] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3713.551504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3713.720431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3715.454511] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3718.052689] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3720.568598] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3723.930694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3724.091248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3726.515604] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3729.168543] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3733.292168] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3743.712693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3743.717118] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3743.723778] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.737868] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3744.740665] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3744.744160] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3745.300922] Lustre: server umount lustre-OST0000 complete [ 3746.657731] Lustre: server umount lustre-MDT0000 complete [ 3748.337667] LustreError: 268009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938051 with bad export cookie 8873881863016166722 [ 3748.342647] LustreError: 268009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3748.344241] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3748.462675] Lustre: server umount lustre-MDT0001 complete [ 3753.606393] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3755.207054] Key type lgssc unregistered [ 3755.390913] LNet: 270971:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3755.396063] LNetError: 270971:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3755.404692] LNet: Removed LNI 192.168.201.109@tcp [ 3755.918129] Key type .llcrypt unregistered [ 3755.920342] Key type ._llcrypt unregistered [ 3767.247146] Key type ._llcrypt registered [ 3767.248692] Key type .llcrypt registered [ 3767.290798] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3767.743291] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3767.752247] alg: No test for adler32 (adler32-zlib) [ 3768.599214] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3768.685785] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3770.263176] Key type lgssc registered [ 3770.697095] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3773.660673] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3773.669217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3774.835160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3776.499613] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3780.044445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3780.213921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3782.079239] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3785.224480] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3787.337294] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3789.860916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3789.995258] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3791.753154] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3793.904749] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3800.230331] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3811.807945] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3811.808783] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3811.812036] LustreError: Skipped 1 previous similar message [ 3811.815340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3811.818654] Lustre: Skipped 1 previous similar message [ 3815.905141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3815.906819] Lustre: Skipped 1 previous similar message [ 3816.975450] Lustre: server umount lustre-OST0000 complete [ 3821.026838] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3821.027624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3821.032360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3821.043804] Lustre: Skipped 2 previous similar messages [ 3824.221184] Lustre: server umount lustre-MDT0000 complete [ 3825.332126] LustreError: 272390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938128 with bad export cookie 4731671009831605512 [ 3825.334741] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3825.335080] LustreError: 272390:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3825.453534] Lustre: server umount lustre-MDT0001 complete [ 3828.782485] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3829.706318] Key type lgssc unregistered [ 3829.817361] LNet: 276691:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3829.819322] LNetError: 276691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3829.828268] LNet: Removed LNI 192.168.201.109@tcp [ 3830.095386] Key type .llcrypt unregistered [ 3830.096385] Key type ._llcrypt unregistered [ 3836.867052] Key type ._llcrypt registered [ 3836.867981] Key type .llcrypt registered [ 3836.900716] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3837.310428] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3837.316027] alg: No test for adler32 (adler32-zlib) [ 3838.153273] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3838.233794] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3839.807166] Key type lgssc registered [ 3840.166425] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3842.618905] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3842.624686] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3843.767547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3845.085223] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3848.122533] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3848.254597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3849.963987] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3853.213599] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3855.302678] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3858.220493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3858.406617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.485108] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3863.279948] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3868.648529] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3873.761708] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3873.762539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3873.763655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3873.764522] LustreError: Skipped 1 previous similar message [ 3873.767393] Lustre: Skipped 1 previous similar message [ 3874.895392] Lustre: server umount lustre-OST0000 complete [ 3878.166949] Lustre: server umount lustre-MDT0000 complete [ 3878.880598] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3878.881447] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3878.881678] LustreError: 277888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3878.881685] LustreError: 277888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3878.883513] Lustre: Skipped 1 previous similar message [ 3879.290192] LustreError: 278661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938182 with bad export cookie 8133078644779975511 [ 3879.292532] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3879.293176] LustreError: 278661:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3879.415528] Lustre: server umount lustre-MDT0001 complete [ 3882.882354] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3883.966186] Key type lgssc unregistered [ 3884.090450] LNet: 281603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3884.092416] LNetError: 281603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3884.100390] LNet: Removed LNI 192.168.201.109@tcp [ 3884.378928] Key type .llcrypt unregistered [ 3884.379888] Key type ._llcrypt unregistered [ 3887.717604] Key type ._llcrypt registered [ 3887.718644] Key type .llcrypt registered [ 3887.750177] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 3892.621570] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 3898.859819] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3899.194750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3899.240474] alg: No test for adler32 (adler32-zlib) [ 3900.095374] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 3900.182669] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3901.759153] Key type lgssc registered [ 3902.303087] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.703848] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.709326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3915.786696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3915.796738] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3915.831710] Lustre: lustre-MDT0000: new disk, initializing [ 3915.851866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3915.876372] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3917.135022] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3922.074528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3922.145800] Lustre: 285129:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3922.178414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3922.181945] Lustre: Skipped 1 previous similar message [ 3922.225808] Lustre: lustre-MDT0001: new disk, initializing [ 3922.273592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3922.293527] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3922.301607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3923.874435] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3927.475829] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3929.223656] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3931.387082] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.501354] Lustre: lustre-OST0000: new disk, initializing [ 3931.503203] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3931.505972] Lustre: 287094:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3931.539077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3933.506759] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3936.273337] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3936.279262] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3936.329189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3937.200653] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3938.497489] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3938.563324] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3939.856369] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3939.914563] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3941.345359] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3941.345796] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3941.347733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3941.348778] LustreError: Skipped 1 previous similar message [ 3941.351710] Lustre: Skipped 1 previous similar message [ 3942.540524] Lustre: server umount lustre-OST0000 complete [ 3946.465033] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3946.466385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3946.470138] Lustre: Skipped 1 previous similar message [ 3946.471558] Lustre: Skipped 1 previous similar message [ 3951.585332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3951.589546] Lustre: Skipped 2 previous similar messages [ 3956.704553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3956.706384] Lustre: Skipped 1 previous similar message [ 3958.239088] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3958.306893] Lustre: server umount lustre-MDT0000 complete [ 3959.620394] LustreError: 285121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938263 with bad export cookie 12973117961389967444 [ 3959.624742] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3959.747939] Lustre: server umount lustre-MDT0001 complete [ 3962.585268] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:44:24 (1786938264) [ 3965.140338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3965.313135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3966.729490] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3969.503157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3969.616816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3970.960167] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3973.228859] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3975.085252] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3977.689417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3977.821521] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3979.714274] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3981.918386] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3995.616361] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3995.622062] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3995.633595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3996.646799] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4001.673977] Lustre: server umount lustre-OST0000 complete [ 4005.344052] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4005.347197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4005.351703] Lustre: Skipped 1 previous similar message [ 4005.353597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4005.355854] Lustre: Skipped 2 previous similar messages [ 4017.119112] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4017.194856] Lustre: server umount lustre-MDT0000 complete [ 4018.368150] LustreError: 289483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938321 with bad export cookie 12973117961389968326 [ 4018.370132] LustreError: 290238:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0000-osp-MDT0001: osp_attr_get update error [0x200000bd2:0x1:0x0]: rc = -5 [ 4018.371316] LustreError: 289483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4018.371498] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4018.477410] Lustre: server umount lustre-MDT0001 complete [ 4021.770869] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4022.695230] Key type lgssc unregistered [ 4022.814380] LNet: 293058:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4022.819167] LNetError: 293058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4022.825311] LNet: Removed LNI 192.168.201.109@tcp [ 4023.195405] Key type .llcrypt unregistered [ 4023.196941] Key type ._llcrypt unregistered [ 4031.711219] Key type ._llcrypt registered [ 4031.712192] Key type .llcrypt registered [ 4031.743924] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4038.091467] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4038.438405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4038.458166] alg: No test for adler32 (adler32-zlib) [ 4039.307552] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 4039.385846] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4040.959204] Key type lgssc registered [ 4041.465856] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4058.905029] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4058.913587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4060.014504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4060.024724] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4060.061190] Lustre: lustre-MDT0000: new disk, initializing [ 4060.083154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4060.089967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4061.716915] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4066.034020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.069339] Lustre: 296853:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4066.089629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4066.091209] Lustre: Skipped 1 previous similar message [ 4066.125029] Lustre: lustre-MDT0001: new disk, initializing [ 4066.153466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4066.166750] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4066.169665] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4067.463182] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4071.331635] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4073.096766] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4075.410496] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4075.509652] Lustre: lustre-OST2710: new disk, initializing [ 4075.512437] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4075.514616] Lustre: 298820:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4075.543176] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4077.408325] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4081.248945] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4083.673496] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4083.761372] Lustre: lustre-OST03e8: new disk, initializing [ 4083.763467] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4083.766158] Lustre: 299871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4083.827772] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4083.852979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4083.858880] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4083.929300] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4083.934689] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4085.837463] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4088.393323] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4091.027216] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4091.111630] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4092.556543] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4092.625389] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4093.890645] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4093.954149] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4095.339541] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4095.399994] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4096.991671] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4096.993640] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4096.996931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.554266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4099.556045] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4099.558122] Lustre: Skipped 1 previous similar message [ 4099.561613] Lustre: Skipped 1 previous similar message [ 4102.626184] Lustre: server umount lustre-MDT0000 complete [ 4103.833710] LustreError: 296891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938407 with bad export cookie 15824473819740856464 [ 4103.837164] LustreError: 296891:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4103.837395] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4103.936797] Lustre: server umount lustre-MDT0001 complete [ 4115.340969] Lustre: server umount lustre-OST2710 complete [ 4125.753282] Lustre: server umount lustre-OST03e8 complete [ 4131.493218] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4134.267150] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4145.314217] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4145.388552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4145.396721] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4145.430416] Lustre: lustre-MDT0000: new disk, initializing [ 4145.448522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.453186] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4145.455218] Lustre: Skipped 1 previous similar message [ 4146.615494] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4150.576068] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4150.611327] Lustre: 305117:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4150.703643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4150.706374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4150.708913] Lustre: Skipped 1 previous similar message [ 4151.959369] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4155.619177] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4157.510445] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4159.808476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4159.909458] Lustre: lustre-OST0000: new disk, initializing [ 4159.910869] Lustre: Skipped 1 previous similar message [ 4159.912282] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4159.914119] Lustre: Skipped 2 previous similar messages [ 4159.915907] Lustre: 307083:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4159.946385] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4159.948109] Lustre: Skipped 1 previous similar message [ 4161.147187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4161.150166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4161.188043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4161.823716] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4165.695106] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4167.091635] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4167.199403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4168.604297] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4168.681499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4171.364596] Lustre: server umount lustre-OST0000 complete [ 4176.351835] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4176.353906] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.356889] Lustre: Skipped 1 previous similar message [ 4176.358334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4181.984761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4181.986582] Lustre: Skipped 2 previous similar messages [ 4183.032438] Lustre: server umount lustre-MDT0000 complete [ 4184.275332] LustreError: 305107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938487 with bad export cookie 15824473819740858046 [ 4184.279474] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4184.280669] LustreError: 305107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4193.801680] Lustre: server umount lustre-MDT0001 complete [ 4196.279184] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:48:18 (1786938498) [ 4196.757375] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4197.323577] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:48:19 (1786938499) [ 4198.265644] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4206.238102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4206.312439] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4206.317947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4206.369896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4207.570194] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4210.138118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4210.166808] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4210.175485] Lustre: 310933:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4211.458659] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4227.041981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4227.106983] Lustre: Denying initial registration attempt from nid 192.168.201.109@tcp, specified as failover [ 4227.109772] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 4227.112163] LustreError: 312381:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 4227.114108] LustreError: 312381:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4227.115813] LustreError: 312381:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4227.135934] Lustre: server umount lustre-OST0000 complete [ 4227.137097] LustreError: 312381:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 4230.623786] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4230.626092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4230.629368] Lustre: Skipped 1 previous similar message [ 4230.631223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4235.216298] Lustre: server umount lustre-MDT0000 complete [ 4236.343080] LustreError: 310924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938539 with bad export cookie 15824473819740858914 [ 4236.346132] LustreError: 310924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4236.346543] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4236.453244] Lustre: server umount lustre-MDT0001 complete [ 4239.648285] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4240.529110] Key type lgssc unregistered [ 4240.645349] LNet: 313509:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4240.647706] LNetError: 313509:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4240.656258] LNet: Removed LNI 192.168.201.109@tcp [ 4240.943449] Key type .llcrypt unregistered [ 4240.944462] Key type ._llcrypt unregistered [ 4244.014349] Key type ._llcrypt registered [ 4244.015343] Key type .llcrypt registered [ 4244.048972] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4244.462479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4244.468414] alg: No test for adler32 (adler32-zlib) [ 4245.311067] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 4245.396917] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4246.975123] Key type lgssc registered [ 4247.279373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4270.552211] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4273.229334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4273.234618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4274.316864] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4274.322935] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4274.373696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4275.578810] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4278.347035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4278.380444] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4278.391153] Lustre: 316244:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4278.498372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4279.873507] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4282.103318] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4283.820417] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4299.042445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4299.128696] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4299.201960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4301.084880] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4303.426234] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4304.143098] Lustre: server umount lustre-OST0000 complete [ 4308.464754] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.466105] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4308.468827] Lustre: Skipped 1 previous similar message [ 4311.450698] Lustre: server umount lustre-MDT0000 complete [ 4312.641168] LustreError: 316238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938616 with bad export cookie 14242245155806450161 [ 4312.646617] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4312.767260] Lustre: server umount lustre-MDT0001 complete [ 4316.166906] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4317.165896] Key type lgssc unregistered [ 4317.295442] LNet: 319809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4317.297699] LNetError: 319809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4317.306467] LNet: Removed LNI 192.168.201.109@tcp [ 4317.609865] Key type .llcrypt unregistered [ 4317.610849] Key type ._llcrypt unregistered [ 4326.426697] Key type ._llcrypt registered [ 4326.427630] Key type .llcrypt registered [ 4326.460323] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4326.862986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4326.875496] alg: No test for adler32 (adler32-zlib) [ 4327.717736] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 4327.799757] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4329.375092] Key type lgssc registered [ 4329.683683] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4332.055316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4332.061373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4333.211326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4334.467356] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4337.315168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4337.446911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4338.763041] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4341.029584] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4342.711319] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4346.786189] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4346.894287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4348.672673] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4350.867192] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4372.449748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4372.451256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4372.453122] Lustre: Skipped 2 previous similar messages [ 4372.454631] Lustre: Skipped 1 previous similar message [ 4375.764303] Lustre: server umount lustre-MDT0000 complete [ 4376.942397] LustreError: 321184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938680 with bad export cookie 15022078236002396305 [ 4376.948447] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4376.949894] LustreError: 321184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4377.568100] LustreError: 323689:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4377.568280] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4377.572538] LustreError: 323689:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4377.577771] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4377.579263] Lustre: Skipped 2 previous similar messages [ 4381.599804] LustreError: 322641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4381.599961] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4381.603973] LustreError: 322641:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4381.607629] Lustre: Skipped 1 previous similar message [ 4382.737981] LustreError: 321198:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.720201] LustreError: 322641:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4386.720229] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4386.726298] Lustre: Skipped 1 previous similar message [ 4391.391117] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4391.464410] Lustre: server umount lustre-MDT0001 complete [ 4393.517150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 4396.198915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4402.143673] LustreError: 320514:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000df6731a4 x1873740961965312/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4402.245546] LustreError: 323032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4402.250265] LustreError: 323032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4402.268944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4402.292653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 4403.564067] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4406.501828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4406.621109] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4407.868093] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4410.153635] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4416.998025] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 4418.834110] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 8 sec [ 4420.569483] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4421.099098] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4422.623889] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4422.627212] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4422.633045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4422.635236] Lustre: Skipped 1 previous similar message [ 4427.234172] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4427.236700] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.432128] Lustre: server umount lustre-OST0000 complete [ 4432.353083] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4432.353298] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.355770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4432.359501] Lustre: Skipped 1 previous similar message [ 4432.362020] Lustre: Skipped 3 previous similar messages [ 4435.733573] Lustre: server umount lustre-MDT0000 complete [ 4436.884060] LustreError: 326445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938740 with bad export cookie 15022078236002403424 [ 4436.886648] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4436.887247] LustreError: 326445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4436.990652] Lustre: server umount lustre-MDT0001 complete [ 4440.554713] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4441.432898] Key type lgssc unregistered [ 4441.552341] LNet: 327486:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4441.554448] LNetError: 327486:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4441.562333] LNet: Removed LNI 192.168.201.109@tcp [ 4441.834133] Key type .llcrypt unregistered [ 4441.836033] Key type ._llcrypt unregistered [ 4450.518331] Key type ._llcrypt registered [ 4450.519289] Key type .llcrypt registered [ 4450.554809] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4451.007849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4451.078135] alg: No test for adler32 (adler32-zlib) [ 4451.933405] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 4452.017779] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4453.591127] Key type lgssc registered [ 4453.939661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4456.347241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4456.353433] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.488114] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4458.699927] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4461.199848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4461.319090] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4462.545343] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4464.721764] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4466.335232] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4471.328943] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.329697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4471.332229] Lustre: Skipped 1 previous similar message [ 4471.336634] Lustre: Skipped 1 previous similar message [ 4473.365298] Lustre: server umount lustre-MDT0000 complete [ 4474.471075] LustreError: 328860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938777 with bad export cookie 13723323506668268201 [ 4474.474131] LustreError: 328860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4474.474222] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4474.575562] Lustre: server umount lustre-MDT0001 complete [ 4479.099159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4479.188210] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4479.197799] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4479.263733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4480.461524] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4483.106152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4483.137348] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4483.147303] Lustre: 331465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4483.235449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4484.492736] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4486.753327] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4488.443691] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4492.963434] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4493.030475] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 4493.034505] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 4493.036951] LustreError: 333347:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 4493.039815] LustreError: 333347:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 4493.042130] LustreError: 333347:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 4493.063527] Lustre: server umount lustre-OST0000 complete [ 4493.064640] LustreError: 333347:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 4497.761850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4497.829884] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4497.886947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4499.576961] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4501.751414] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4503.582332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 4504.309093] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4504.339270] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4504.376360] Lustre: lustre-OST0001: new disk, initializing [ 4504.377879] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4504.379958] Lustre: 334886:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4506.314097] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4508.672044] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4509.687606] Lustre: server umount lustre-OST0001 complete [ 4511.726513] LustreError: 333814:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4511.728633] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4511.729498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4511.734202] LustreError: 333814:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4511.737298] Lustre: Skipped 1 previous similar message [ 4516.833404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4516.834413] LustreError: 335676:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4516.835950] Lustre: Skipped 2 previous similar messages [ 4516.841069] LustreError: 335676:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4517.000291] Lustre: server umount lustre-OST0000 complete [ 4519.391748] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4519.393777] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4519.397374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4521.953093] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4524.251169] Lustre: server umount lustre-MDT0000 complete [ 4525.461143] LustreError: 331501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938828 with bad export cookie 13723323506668268740 [ 4525.464168] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4525.466992] LustreError: 331501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4525.586541] Lustre: server umount lustre-MDT0001 complete [ 4529.042191] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4529.995945] Key type lgssc unregistered [ 4530.112383] LNet: 336839:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4530.114550] LNetError: 336839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4530.121261] LNet: Removed LNI 192.168.201.109@tcp [ 4530.431578] Key type .llcrypt unregistered [ 4530.432607] Key type ._llcrypt unregistered [ 4565.487948] Key type ._llcrypt registered [ 4565.489566] Key type .llcrypt registered [ 4565.534309] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4571.582309] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4571.904283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4571.933530] alg: No test for adler32 (adler32-zlib) [ 4572.768473] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 4572.840382] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4574.415127] Key type lgssc registered [ 4574.765625] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4585.916067] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4585.922676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4587.007570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4587.019237] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4587.056522] Lustre: lustre-MDT0000: new disk, initializing [ 4587.077048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4587.082552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4588.512851] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4592.700830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4592.745612] Lustre: 341740:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4592.772251] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4592.774645] Lustre: Skipped 1 previous similar message [ 4592.815526] Lustre: lustre-MDT0001: new disk, initializing [ 4592.852892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4592.865746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4592.868916] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4594.252515] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4598.129332] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4600.000244] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4602.283658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4602.388254] Lustre: lustre-OST0000: new disk, initializing [ 4602.391073] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4602.394740] Lustre: 343708:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4602.424022] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4604.323113] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4608.139695] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4609.594986] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4612.602288] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4612.607421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4612.640370] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4612.733907] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 4614.196911] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4614.285229] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4617.696611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4617.697801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4617.699830] LustreError: Skipped 1 previous similar message [ 4617.704496] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4617.708919] Lustre: Skipped 1 previous similar message [ 4621.077172] Lustre: server umount lustre-OST0000 complete [ 4622.816384] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4622.817056] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4622.819779] Lustre: Skipped 2 previous similar messages [ 4627.936368] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4627.938500] Lustre: Skipped 2 previous similar messages [ 4628.500808] Lustre: server umount lustre-MDT0000 complete [ 4629.686178] LustreError: 343707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786938933 with bad export cookie 2495202253551122413 [ 4629.689650] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4629.691344] LustreError: 343707:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4629.810549] Lustre: server umount lustre-MDT0001 complete [ 4632.584721] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:55:34 (1786938934) [ 4635.026613] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:55:37 (1786938937) [ 4637.857380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4638.044494] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4639.318215] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4642.126168] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4642.243507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4643.699113] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4646.203852] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4648.080851] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4650.386715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4650.515629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4652.556524] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4655.040641] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4668.828976] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 4669.357602] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4669.880420] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 4670.426274] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 4671.077737] Lustre: Failing over lustre-MDT0000 [ 4671.242571] Lustre: server umount lustre-MDT0000 complete [ 4672.993933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4672.994888] LustreError: 348970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4672.997331] Lustre: Skipped 2 previous similar messages [ 4672.997583] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4673.001930] LustreError: 348970:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4675.610155] LustreError: 348970:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4678.112046] LustreError: 347210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4678.117012] LustreError: 347210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4680.722398] LustreError: 347211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.054863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4684.096948] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4684.187274] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4684.209331] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4685.693117] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4685.842127] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4689.385646] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4689.397653] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4689.417284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4691.218407] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4691.723740] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4693.576625] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 4694.220267] Lustre: Failing over lustre-MDT0000 [ 4694.485982] Lustre: server umount lustre-MDT0000 complete [ 4694.498745] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4694.499977] LustreError: 346482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4694.504478] Lustre: Skipped 2 previous similar messages [ 4694.508872] LustreError: 346482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4694.511159] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4704.736860] LustreError: 347210:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4704.742778] LustreError: 347210:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4707.369245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4707.414054] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4707.546399] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4708.813940] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4710.436239] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4712.940380] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4712.943144] Lustre: Skipped 2 previous similar messages [ 4712.950735] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 4712.966866] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 4714.638634] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4715.116593] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 4718.053320] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4718.055116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4718.055656] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4718.056753] Lustre: Skipped 2 previous similar messages [ 4718.058390] Lustre: Skipped 1 previous similar message [ 4723.169552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4723.172503] Lustre: Skipped 3 previous similar messages [ 4723.482592] Lustre: server umount lustre-MDT0000 complete [ 4724.743912] LustreError: 346463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939028 with bad export cookie 2495202253551125066 [ 4724.749395] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4724.871893] Lustre: server umount lustre-MDT0001 complete [ 4736.530302] Lustre: server umount lustre-OST0000 complete [ 4740.958692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4741.095922] LustreError: 352829:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4741.100427] LustreError: 352829:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4741.116459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4741.118671] Lustre: Skipped 1 previous similar message [ 4742.261774] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4744.834055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4746.096418] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4748.283966] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4749.978718] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4752.017181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4753.770874] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4755.859678] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4766.438115] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 4775.904422] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4775.906427] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4775.910032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4778.570299] Lustre: server umount lustre-OST0000 complete [ 4785.816990] Lustre: server umount lustre-MDT0000 complete [ 4786.966105] LustreError: 352810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939090 with bad export cookie 2495202253551125808 [ 4786.968938] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4786.969804] LustreError: 352810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4787.075975] Lustre: server umount lustre-MDT0001 complete [ 4790.654021] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4791.633543] Key type lgssc unregistered [ 4791.759440] LNet: 356382:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4791.762697] LNetError: 356382:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4791.770264] LNet: Removed LNI 192.168.201.109@tcp [ 4792.069494] Key type .llcrypt unregistered [ 4792.070566] Key type ._llcrypt unregistered [ 4801.347221] Key type ._llcrypt registered [ 4801.348216] Key type .llcrypt registered [ 4801.384537] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4801.747123] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4801.796092] alg: No test for adler32 (adler32-zlib) [ 4802.642820] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 4802.729594] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4804.303103] Key type lgssc registered [ 4804.642027] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4807.063196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4807.068778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4808.208322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4809.409069] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4811.975158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.084034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4813.285531] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4815.541940] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4817.232252] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4819.525173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4819.636717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4821.605401] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4823.813881] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4832.930588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 4838.369206] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4838.370682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4838.374029] Lustre: Skipped 2 previous similar messages [ 4838.379139] Lustre: Skipped 2 previous similar messages [ 4843.489236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4843.491620] Lustre: Skipped 2 previous similar messages [ 4844.312891] Lustre: server umount lustre-MDT0000 complete [ 4845.490238] LustreError: 360476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939148 with bad export cookie 7836744349676280209 [ 4845.494125] LustreError: 360476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4845.495973] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4845.625544] Lustre: server umount lustre-MDT0001 complete [ 4857.114086] Lustre: server umount lustre-OST0000 complete [ 4861.153314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4861.313837] LustreError: 361606:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4861.318380] LustreError: 361606:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4861.339462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4862.606980] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4865.305070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4865.435506] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4866.703269] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4868.946712] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4870.696480] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4872.886705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4874.676261] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4876.859882] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4886.310289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 4896.224501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4896.228851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.235883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4900.977526] Lustre: server umount lustre-MDT0000 complete [ 4901.857751] LustreError: 364095:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.863217] LustreError: 364095:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4902.298696] LustreError: 362340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939205 with bad export cookie 7836744349676281308 [ 4902.304920] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4902.425665] Lustre: server umount lustre-MDT0001 complete [ 4913.948050] Lustre: server umount lustre-OST0000 complete [ 4918.945407] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4919.097408] LustreError: 365540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4919.122789] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4919.124917] Lustre: Skipped 1 previous similar message [ 4920.510832] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4923.356300] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4924.727522] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4926.994640] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4928.739480] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4930.826702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4932.726752] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4935.013448] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4945.254579] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 4948.447796] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4948.450228] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4948.453339] Lustre: Skipped 2 previous similar messages [ 4948.454863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4948.456361] Lustre: Skipped 2 previous similar messages [ 4950.496923] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4950.498028] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4952.715413] Lustre: server umount lustre-OST0000 complete [ 4954.079889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4954.082294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4954.083868] Lustre: Skipped 2 previous similar messages [ 4955.617147] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4955.620885] Lustre: Skipped 1 previous similar message [ 4960.027565] Lustre: server umount lustre-MDT0000 complete [ 4960.736783] LustreError: 365559:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4960.746600] LustreError: 365559:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4961.631973] LustreError: 365522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939265 with bad export cookie 7836744349676282316 [ 4961.635225] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4961.635330] LustreError: 365522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 4961.746970] Lustre: server umount lustre-MDT0001 complete [ 4965.372962] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4966.677191] Key type lgssc unregistered [ 4966.824567] LNet: 369106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4966.828728] LNetError: 369106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4966.838645] LNet: Removed LNI 192.168.201.109@tcp [ 4967.286739] Key type .llcrypt unregistered [ 4967.288835] Key type ._llcrypt unregistered [ 4977.583329] Key type ._llcrypt registered [ 4977.584342] Key type .llcrypt registered [ 4977.614110] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4978.034244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.044259] alg: No test for adler32 (adler32-zlib) [ 4978.881952] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 4978.960474] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4980.535191] Key type lgssc registered [ 4981.110830] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4983.791497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4983.796910] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4983.799470] LustreError: 370525:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 4983.808798] LustreError: 370525:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 4983.810786] LustreError: 370525:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 4983.812833] LustreError: 370525:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 4983.815099] LustreError: 370525:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4986.337457] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4986.339777] LustreError: 370921:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 4986.347749] LustreError: 370921:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 4986.349595] LustreError: 370921:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 4986.351509] LustreError: 370921:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 4986.353832] LustreError: 370921:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 4993.631372] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4994.519815] Key type lgssc unregistered [ 4994.632294] LNet: 371925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4994.634342] LNetError: 371925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4994.642291] LNet: Removed LNI 192.168.201.109@tcp [ 4994.905285] Key type .llcrypt unregistered [ 4994.906284] Key type ._llcrypt unregistered [ 5001.995721] Key type ._llcrypt registered [ 5001.997140] Key type .llcrypt registered [ 5002.045703] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5009.584690] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5009.868570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5009.901055] alg: No test for adler32 (adler32-zlib) [ 5010.759815] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5010.850734] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5012.439185] Key type lgssc registered [ 5013.033840] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5027.219361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5027.229107] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5028.319963] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5028.329798] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5028.363569] Lustre: lustre-MDT0000: new disk, initializing [ 5028.382756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5028.387608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5029.568216] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5033.781799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5033.817879] Lustre: 375055:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5033.837992] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5033.839744] Lustre: Skipped 1 previous similar message [ 5033.872586] Lustre: lustre-MDT0001: new disk, initializing [ 5033.900899] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5033.911032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5033.915609] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5035.120643] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5039.010613] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5040.990069] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5043.322132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5043.428436] Lustre: lustre-OST0000: new disk, initializing [ 5043.429901] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5043.432128] Lustre: 377022:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5043.464131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5045.218278] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5048.944814] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5050.404956] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5053.437075] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5053.439888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5053.481237] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5053.558201] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 5054.778861] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5054.838615] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5058.529518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5058.530863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5058.533154] LustreError: Skipped 1 previous similar message [ 5058.537918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5058.541132] Lustre: Skipped 1 previous similar message [ 5061.514780] Lustre: server umount lustre-OST0000 complete [ 5063.648633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5063.649305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5063.651826] Lustre: Skipped 2 previous similar messages [ 5068.768313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5068.769996] Lustre: Skipped 2 previous similar messages [ 5068.829884] Lustre: server umount lustre-MDT0000 complete [ 5069.998117] LustreError: 375849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939373 with bad export cookie 17473135218016704094 [ 5070.000692] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5070.004139] LustreError: 375849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5070.120391] Lustre: server umount lustre-MDT0001 complete [ 5072.964590] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 5073.651892] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:02:55 (1786939375) [ 5075.732633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.897364] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5077.096681] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5079.665054] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5081.232427] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5083.583692] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5085.278660] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5087.383536] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5087.491147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5087.492973] Lustre: Skipped 1 previous similar message [ 5089.221340] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5091.454017] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5093.552833] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5093.619501] Lustre: lustre-OST0001: new disk, initializing [ 5093.621037] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5093.623252] Lustre: 382333:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5095.305171] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5095.801925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5095.804929] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5095.837284] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5098.900963] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5101.027335] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5101.028232] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5101.032447] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5101.035848] Lustre: Skipped 1 previous similar message [ 5105.176444] Lustre: lustre-OST0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 5105.178225] Lustre: Skipped 2 previous similar messages [ 5106.746404] Lustre: server umount lustre-OST0001 complete [ 5107.170526] LustreError: 381319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5107.175241] LustreError: 381319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5112.296477] LustreError: 381320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5112.306980] LustreError: 381320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5117.416313] LustreError: 381747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5117.425436] LustreError: 381747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5120.480374] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5120.485343] LustreError: Skipped 1 previous similar message [ 5120.488259] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5120.496121] Lustre: Skipped 1 previous similar message [ 5120.499434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5122.529925] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5122.536265] LustreError: 381318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.546459] LustreError: 381318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5127.652004] LustreError: 381747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5127.659341] LustreError: 381747:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5133.279153] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5133.365235] Lustre: server umount lustre-OST0000 complete [ 5136.352450] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5136.357123] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5136.364320] Lustre: Skipped 1 previous similar message [ 5137.889583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5137.894140] Lustre: Skipped 8 previous similar messages [ 5141.303163] Lustre: server umount lustre-MDT0000 complete [ 5142.640495] LustreError: 379479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939446 with bad export cookie 17473135218016704962 [ 5142.644374] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5142.645609] LustreError: 379479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5148.872382] Lustre: server umount lustre-MDT0001 complete [ 5153.763110] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5155.013649] Key type lgssc unregistered [ 5155.142469] LNet: 384321:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5155.146670] LNetError: 384321:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5155.155469] LNet: Removed LNI 192.168.201.109@tcp [ 5155.488436] Key type .llcrypt unregistered [ 5155.490462] Key type ._llcrypt unregistered [ 5179.375505] Key type ._llcrypt registered [ 5179.376975] Key type .llcrypt registered [ 5179.435868] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5179.837283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5179.871605] alg: No test for adler32 (adler32-zlib) [ 5180.756928] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5180.874826] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5182.463172] Key type lgssc registered [ 5183.071829] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5186.835738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5186.846574] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5187.970726] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5187.983421] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5188.067299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5189.838516] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5193.717141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5193.748888] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5193.759407] Lustre: 387095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5193.869337] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5195.134378] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5197.444751] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5199.353504] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5204.448980] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5204.452395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.458644] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.961980] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.977752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5206.213767] Lustre: server umount lustre-MDT0000 complete [ 5207.813671] LustreError: 387089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939511 with bad export cookie 13214234134898264017 [ 5207.816425] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5207.817976] LustreError: 387089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5207.986300] Lustre: server umount lustre-MDT0001 complete [ 5211.755412] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:05:13 (1786939513) [ 5215.191841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5215.527144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5218.551716] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5220.831967] LustreError: 389607:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5220.850025] LustreError: 389607:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5224.383026] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.594149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5227.567337] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5233.287708] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5236.037643] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5238.205177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5238.268967] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5238.325644] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5240.243253] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5242.937702] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5247.284950] Lustre: Permanently deactivating lustre-OST0000 [ 5247.295332] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5257.203756] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5257.912756] LustreError: 392735:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 5257.917235] LustreError: 392735:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5257.921029] LustreError: 392735:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5258.622797] LustreError: 392784:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5258.626555] LustreError: 392784:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5265.422196] Lustre: server umount lustre-OST0000 complete [ 5266.311758] LustreError: 393034:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [ 5266.315220] LustreError: 393034:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 5269.472168] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5269.473086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5269.475285] Lustre: Skipped 1 previous similar message [ 5274.594939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5274.597774] Lustre: Skipped 2 previous similar messages [ 5279.712299] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5279.714603] Lustre: Skipped 1 previous similar message [ 5281.247102] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5281.306853] Lustre: server umount lustre-MDT0000 complete [ 5282.538196] LustreError: 389586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939586 with bad export cookie 13214234134898264584 [ 5282.541647] LustreError: 389586:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5282.541914] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5293.066148] Lustre: server umount lustre-MDT0001 complete [ 5295.861181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5297.170482] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5298.042245] LustreError: 394192:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 5299.940464] Lustre: 394531:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.109@tcp [ 5300.607746] Lustre: server umount MGS complete [ 5304.601461] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.749940] LustreError: 395254:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5304.755234] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5304.758991] Lustre: Skipped 1 previous similar message [ 5304.780469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.162864] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5309.921753] LustreError: 395255:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5311.419371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5311.593298] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5314.887430] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5320.129962] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5324.301423] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5325.813534] Lustre: Permanently reactivating lustre-OST0000 [ 5325.821365] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5325.824198] Lustre: Skipped 2 previous similar messages [ 5351.923535] LustreError: 397032:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 5354.734031] hrtimer: interrupt took 9152368 ns [ 5358.160288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.426723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5358.434267] Lustre: Skipped 1 previous similar message [ 5362.693311] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5366.839933] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5370.848512] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5370.851056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5370.857306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5371.893835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5375.295654] Lustre: server umount lustre-OST0000 complete [ 5378.528400] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5378.531692] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.535957] Lustre: Skipped 1 previous similar message [ 5383.470909] Lustre: server umount lustre-MDT0000 complete [ 5385.411234] LustreError: 398863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939688 with bad export cookie 13214234134898265634 [ 5385.411693] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5385.416742] LustreError: 398863:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5385.591155] Lustre: server umount lustre-MDT0001 complete [ 5391.224547] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5392.861798] Key type lgssc unregistered [ 5393.066511] LNet: 399617:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5393.070072] LNetError: 399617:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5393.076644] LNet: Removed LNI 192.168.201.109@tcp [ 5393.559652] Key type .llcrypt unregistered [ 5393.561168] Key type ._llcrypt unregistered [ 5403.487770] Key type ._llcrypt registered [ 5403.489914] Key type .llcrypt registered [ 5403.535745] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5412.405418] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5412.950465] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5413.043388] alg: No test for adler32 (adler32-zlib) [ 5413.988897] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5414.122360] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5415.721599] Key type lgssc registered [ 5416.441139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5436.339660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5436.358324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5437.465774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5437.486841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5437.540346] Lustre: lustre-MDT0000: new disk, initializing [ 5437.589712] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5437.598896] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5439.574423] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5445.704977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5445.758603] Lustre: 402749:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5445.789530] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5445.793903] Lustre: Skipped 1 previous similar message [ 5445.832957] Lustre: lustre-MDT0001: new disk, initializing [ 5445.896800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5445.917403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5445.922767] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5448.011277] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5452.532752] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5454.867599] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5457.889127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5458.042080] Lustre: lustre-OST0000: new disk, initializing [ 5458.044609] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5458.047716] Lustre: 404716:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5458.088457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5460.616552] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5465.511854] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5468.184301] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5468.197872] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5468.208815] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5468.265342] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5469.371594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 5471.703661] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5471.806558] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5473.252176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5473.254570] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5473.259948] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5473.262917] Lustre: Skipped 1 previous similar message [ 5474.791996] Lustre: server umount lustre-OST0000 complete [ 5476.833397] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5476.840425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5476.848669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5476.854274] Lustre: Skipped 1 previous similar message [ 5478.368586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5478.368825] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5483.491587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5483.495252] Lustre: Skipped 2 previous similar messages [ 5488.610259] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5488.613620] Lustre: Skipped 1 previous similar message [ 5491.167138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5491.272699] Lustre: server umount lustre-MDT0000 complete [ 5493.431502] LustreError: 402742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939796 with bad export cookie 6654411333954185061 [ 5493.432675] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5493.437064] LustreError: 402742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5493.621156] Lustre: server umount lustre-MDT0001 complete [ 5498.470886] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 5499.279709] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:10:01 (1786939801) [ 5503.109116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.418724] LustreError: 407184:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5503.427898] LustreError: 407184:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5503.459225] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5505.678184] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5508.576936] LustreError: 407185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5509.980618] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5510.184612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5511.902373] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5515.051761] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5517.361285] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5520.589251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5520.749760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5523.537121] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5527.404633] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5528.549133] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 5530.326952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 5538.272114] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5538.275918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5538.281750] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5540.322389] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5543.130115] Lustre: server umount lustre-OST0000 complete [ 5545.441794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5545.446132] Lustre: Skipped 2 previous similar messages [ 5550.558269] Lustre: server umount lustre-MDT0000 complete [ 5550.563322] LustreError: 407185:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.570308] LustreError: 407185:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5551.818122] LustreError: 407166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939855 with bad export cookie 6654411333954185936 [ 5551.820042] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5551.822188] LustreError: 407166:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5551.940116] Lustre: server umount lustre-MDT0001 complete [ 5555.763896] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5556.963395] Key type lgssc unregistered [ 5557.088429] LNet: 410907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5557.091794] LNetError: 410907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5557.100485] LNet: Removed LNI 192.168.201.109@tcp [ 5557.410974] Key type .llcrypt unregistered [ 5557.412196] Key type ._llcrypt unregistered [ 5565.818758] Key type ._llcrypt registered [ 5565.819912] Key type .llcrypt registered [ 5565.857993] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5565.982384] Key type .llcrypt unregistered [ 5565.983680] Key type ._llcrypt unregistered [ 5573.061312] Key type ._llcrypt registered [ 5573.062393] Key type .llcrypt registered [ 5573.098420] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5573.386623] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5573.401651] alg: No test for adler32 (adler32-zlib) [ 5574.254337] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5574.344553] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5575.919157] Key type lgssc registered [ 5576.274990] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5578.739818] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5578.745547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5579.908324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5581.311791] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5584.307296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5584.432137] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5586.358354] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5588.692394] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5589.472692] LustreError: 413097:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5589.479024] LustreError: 413097:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5590.496520] LustreError: 413098:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5590.501526] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5590.505310] LustreError: 413098:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5590.951604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5591.075325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5592.415252] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5600.735807] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5600.737811] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5600.742319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.249794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5601.250644] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5601.253739] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5605.388785] Lustre: server umount lustre-OST0000 complete [ 5609.953190] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 5609.955666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5609.959945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5609.961984] Lustre: Skipped 1 previous similar message [ 5612.699114] Lustre: server umount lustre-MDT0000 complete [ 5613.987043] LustreError: 414838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786939917 with bad export cookie 7670731733974040377 [ 5613.990839] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5613.991297] LustreError: 414838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5614.111467] Lustre: server umount lustre-MDT0001 complete [ 5617.722713] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5618.725664] Key type lgssc unregistered [ 5618.850437] LNet: 416248:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5618.852936] LNetError: 416248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5618.870298] LNet: Removed LNI 192.168.201.109@tcp [ 5619.178217] Key type .llcrypt unregistered [ 5619.179838] Key type ._llcrypt unregistered [ 5628.053432] Key type ._llcrypt registered [ 5628.054548] Key type .llcrypt registered [ 5628.090352] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5628.487389] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5628.494798] alg: No test for adler32 (adler32-zlib) [ 5629.342642] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5629.424303] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5630.999132] Key type lgssc registered [ 5631.327348] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5633.683284] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5633.689138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5649.119372] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 5654.239233] LustreError: 417614:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 5680.863175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5682.869147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5685.250809] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5687.561762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5687.723512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.749022] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5689.008754] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5691.657945] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5691.767481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5693.055162] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5695.369522] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5697.125342] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5701.090673] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5701.091376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5701.096520] Lustre: Skipped 1 previous similar message [ 5705.420179] Lustre: server umount lustre-OST0000 complete [ 5707.231823] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5707.234440] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5707.239743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5707.242074] Lustre: Skipped 1 previous similar message [ 5711.328096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5711.328555] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.719768] Lustre: server umount lustre-MDT0000 complete [ 5713.900431] LustreError: 420145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940017 with bad export cookie 7447210837183398733 [ 5713.901616] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5713.904370] LustreError: 420145:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5714.015574] Lustre: server umount lustre-MDT0001 complete [ 5717.505458] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5718.481511] Key type lgssc unregistered [ 5718.607334] LNet: 421187:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5718.610229] LNetError: 421187:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5718.620296] LNet: Removed LNI 192.168.201.109@tcp [ 5718.915320] Key type .llcrypt unregistered [ 5718.916713] Key type ._llcrypt unregistered [ 5727.719737] Key type ._llcrypt registered [ 5727.720995] Key type .llcrypt registered [ 5727.758948] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5728.128850] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5728.164099] alg: No test for adler32 (adler32-zlib) [ 5729.024440] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5729.115450] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5730.703079] Key type lgssc registered [ 5731.089231] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5733.785218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5733.790588] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.966843] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5736.411985] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5739.229460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.352784] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5740.699414] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5742.996596] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5744.747745] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5746.919959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5747.028063] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5748.924085] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5751.089284] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5761.384967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5765.087807] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5765.090152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5765.094137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5766.624224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5766.625909] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5766.626815] Lustre: Skipped 1 previous similar message [ 5766.632166] Lustre: Skipped 1 previous similar message [ 5768.465789] Lustre: server umount lustre-MDT0000 complete [ 5769.169236] LustreError: 422559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.9@tcp arrived at 1786940072 with bad export cookie 2370071966166875995 [ 5769.173971] LustreError: 422559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5769.804130] LustreError: 422561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940073 with bad export cookie 2370071966166875344 [ 5769.807990] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5769.808031] LustreError: 422561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5771.744798] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5771.744837] LustreError: 425071:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5771.750018] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5771.755045] LustreError: 425071:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5771.756966] Lustre: Skipped 2 previous similar messages [ 5774.751887] LustreError: 422580:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5774.752201] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5774.756062] LustreError: 422580:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5775.949329] Lustre: server umount lustre-OST0000 complete [ 5778.157447] Lustre: server umount lustre-MDT0001 complete [ 5781.465852] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5782.428399] Key type lgssc unregistered [ 5782.543349] LNet: 426289:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5782.545537] LNetError: 426289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5782.551290] LNet: Removed LNI 192.168.201.109@tcp [ 5782.836227] Key type .llcrypt unregistered [ 5782.837302] Key type ._llcrypt unregistered [ 5791.409260] Key type ._llcrypt registered [ 5791.410199] Key type .llcrypt registered [ 5791.440539] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5791.779200] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5791.796507] alg: No test for adler32 (adler32-zlib) [ 5792.648785] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5792.725714] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5794.303164] Key type lgssc registered [ 5794.644922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5797.033769] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5797.039244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5798.174810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5799.489569] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5802.256467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5802.370309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5803.672930] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5805.972224] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5807.616596] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5809.742652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.847187] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5811.686100] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5813.939657] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5823.141613] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5828.064092] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5828.066637] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.070818] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5828.577409] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5828.583327] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5829.659093] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 5831.568723] Lustre: server umount lustre-MDT0001 complete [ 5833.696417] LustreError: 427680:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5833.696475] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5833.696781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5833.697275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.701756] LustreError: 427680:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5838.816831] LustreError: 427681:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5838.817692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5839.249446] Lustre: server umount lustre-OST0000 complete [ 5840.494662] Lustre: server umount lustre-MDT0000 complete [ 5844.785157] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5845.783451] Key type lgssc unregistered [ 5845.909417] LNet: 431387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5845.912034] LNetError: 431387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5845.918261] LNet: Removed LNI 192.168.201.109@tcp [ 5846.215193] Key type .llcrypt unregistered [ 5846.216200] Key type ._llcrypt unregistered [ 5853.788411] Key type ._llcrypt registered [ 5853.789354] Key type .llcrypt registered [ 5853.825690] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5859.451684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5859.770854] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5859.805069] alg: No test for adler32 (adler32-zlib) [ 5860.661682] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5860.744649] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5862.319164] Key type lgssc registered [ 5862.679162] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5873.788176] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5878.517861] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5878.524235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.601815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5879.611430] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5879.644263] Lustre: lustre-MDT0000: new disk, initializing [ 5879.662219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5879.667827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5880.923595] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5886.407912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5886.522505] Lustre: lustre-OST0000: new disk, initializing [ 5886.523911] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5886.525367] Lustre: Skipped 1 previous similar message [ 5886.527085] Lustre: 436388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5886.552154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5888.493264] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5891.954447] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5893.113577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 5893.117165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 5893.124156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 5902.450646] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5902.496904] Lustre: 435373:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5902.528042] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5902.558754] Lustre: lustre-MDT0001: new disk, initializing [ 5902.598431] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5902.610258] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 5902.614196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 5903.803467] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5907.255799] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 5907.317723] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 5908.510017] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 5908.578405] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 5913.570578] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.571260] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5913.573743] Lustre: Skipped 1 previous similar message [ 5913.576938] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 5914.136701] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 5914.138930] Lustre: Skipped 1 previous similar message [ 5916.177127] Lustre: server umount lustre-MDT0001 complete [ 5923.809722] LustreError: 435378:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.815857] LustreError: 435378:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 5928.928968] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5928.929123] LustreError: 435382:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.929402] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5928.930234] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5934.049663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5934.050051] LustreError: 435383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5934.490140] Lustre: server umount lustre-OST0000 complete [ 5941.881300] Lustre: server umount lustre-MDT0000 complete [ 5946.128656] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5947.080240] Key type lgssc unregistered [ 5947.195342] LNet: 440136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5947.197555] LNetError: 440136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5947.203228] LNet: Removed LNI 192.168.201.109@tcp [ 5947.503130] Key type .llcrypt unregistered [ 5947.504567] Key type ._llcrypt unregistered [ 5959.192606] Key type ._llcrypt registered [ 5959.193553] Key type .llcrypt registered [ 5959.229171] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5959.341366] Key type .llcrypt unregistered [ 5959.342353] Key type ._llcrypt unregistered [ 5965.451343] Key type ._llcrypt registered [ 5965.452553] Key type .llcrypt registered [ 5965.486227] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5965.766510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5965.791106] alg: No test for adler32 (adler32-zlib) [ 5966.650784] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 5966.730456] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 5968.311126] Key type lgssc registered [ 5968.641192] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5976.907870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5976.913452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5977.984805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5977.994690] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5978.034083] Lustre: lustre-MDT0000: new disk, initializing [ 5978.054587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5978.063428] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5979.345202] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5983.298228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.338229] Lustre: 443484:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5983.360358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5983.362536] Lustre: Skipped 1 previous similar message [ 5983.396710] Lustre: lustre-MDT0001: new disk, initializing [ 5983.428175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5983.441338] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5983.444662] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5984.696793] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5988.230197] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5989.854255] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5991.956517] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5992.057707] Lustre: lustre-OST0000: new disk, initializing [ 5992.059592] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5992.062555] Lustre: 445455:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5992.088668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5993.813674] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5996.536067] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5996.539807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5996.572679] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5997.264097] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6001.632627] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6001.634580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6001.635806] Lustre: Skipped 2 previous similar messages [ 6001.638309] Lustre: Skipped 2 previous similar messages [ 6005.588765] Lustre: server umount lustre-MDT0000 complete [ 6006.716067] LustreError: 445454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940310 with bad export cookie 9249980460221417066 [ 6006.719050] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6006.720541] LustreError: 445454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6006.752239] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6006.757106] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6006.840410] Lustre: server umount lustre-MDT0001 complete [ 6014.105343] Lustre: server umount lustre-OST0000 complete [ 6017.147079] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:18:39 (1786940319) [ 6019.571774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6019.722664] LustreError: 447482:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6019.726847] LustreError: 447482:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6019.743844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.063530] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6023.609920] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6024.926480] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6027.054529] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6028.850891] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6043.938159] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6044.011798] Lustre: Found index 0 for lustre-OST0000, updating log [ 6044.075587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.078297] Lustre: Skipped 1 previous similar message [ 6045.095101] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 6046.089498] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6048.542895] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6050.271996] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6050.275345] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6050.281122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6054.369369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6054.372540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.376134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6055.827289] Lustre: server umount lustre-OST0000 complete [ 6069.727626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6069.730411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6069.734653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6069.736601] Lustre: Skipped 2 previous similar messages [ 6074.580830] Lustre: server umount lustre-MDT0000 complete [ 6075.692336] LustreError: 447464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786940379 with bad export cookie 9249980460221418445 [ 6075.695097] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6075.696940] LustreError: 447464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6075.801349] Lustre: server umount lustre-MDT0001 complete [ 6078.392897] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:19:40 (1786940380) [ 6115.231233] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp [ 6678.336404] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6678.684028] LustreError: 453886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6678.766395] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6682.309200] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6684.133716] LustreError: 453886:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6689.248885] LustreError: 453887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6689.929733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6690.230757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6694.026285] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6699.931652] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6704.595083] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6723.567584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6723.764256] Lustre: Found index 0 for lustre-OST0000, updating log [ 6723.962920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6728.071294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 6729.371588] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6729.696787] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp [ 6738.141682] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6741.670363] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6741.677986] Lustre: Skipped 1 previous similar message [ 6741.682114] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6741.685701] Lustre: Skipped 2 previous similar messages [ 6747.220451] Lustre: server umount lustre-OST0000 complete [ 6754.209833] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6754.244040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6756.332442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6756.338159] Lustre: Skipped 4 previous similar messages [ 6760.041909] Lustre: server umount lustre-MDT0000 complete [ 6760.422716] LustreError: 453887:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6760.434229] LustreError: 453887:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6764.586597] LustreError: 454614:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6764.609798] LustreError: 454614:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6767.912942] LustreError: 453867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941071 with bad export cookie 9249980460221419509 [ 6767.925301] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6767.948803] LustreError: 453867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6768.718915] Lustre: server umount lustre-MDT0001 complete [ 6793.316673] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp [ 6801.167789] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 6812.775905] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6848.674785] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 00:32:29 (1786941149) [ 6866.021900] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6868.893384] Key type lgssc unregistered [ 6869.241808] LNet: 460499:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6869.246295] LNetError: 460499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6985.005648] LNet: Removed LNI 192.168.201.109@tcp [ 6985.587801] Key type .llcrypt unregistered [ 6985.589934] Key type ._llcrypt unregistered [ 6997.678244] Key type ._llcrypt registered [ 6997.680474] Key type .llcrypt registered [ 6997.773755] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6998.629485] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6998.657066] alg: No test for adler32 (adler32-zlib) [ 6999.627401] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 6999.809634] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7001.441708] Key type lgssc registered [ 7002.257665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7005.474971] LNet: Added LNI 192.168.201.109@tcp100 [8/256/0/180] [ 7006.059481] LNet: Added LNI 192.168.201.109@tcp101 [8/256/0/180] [ 7006.687700] LNet: Added LNI 192.168.201.109@tcp102 [8/256/0/180] [ 7007.356762] LNet: Added LNI 192.168.201.109@tcp103 [8/256/0/180] [ 7008.016652] LNet: Added LNI 192.168.201.109@tcp104 [8/256/0/180] [ 7008.667571] LNet: Added LNI 192.168.201.109@tcp105 [8/256/0/180] [ 7009.309943] LNet: Added LNI 192.168.201.109@tcp106 [8/256/0/180] [ 7009.997484] LNet: Added LNI 192.168.201.109@tcp107 [8/256/0/180] [ 7010.684358] LNet: Added LNI 192.168.201.109@tcp108 [8/256/0/180] [ 7011.318628] LNet: Added LNI 192.168.201.109@tcp109 [8/256/0/180] [ 7012.019379] LNet: Added LNI 192.168.201.109@tcp110 [8/256/0/180] [ 7012.757272] LNet: Added LNI 192.168.201.109@tcp111 [8/256/0/180] [ 7013.500900] LNet: Added LNI 192.168.201.109@tcp112 [8/256/0/180] [ 7014.229505] LNet: Added LNI 192.168.201.109@tcp113 [8/256/0/180] [ 7014.773052] LNet: Added LNI 192.168.201.109@tcp114 [8/256/0/180] [ 7015.405400] LNet: Added LNI 192.168.201.109@tcp115 [8/256/0/180] [ 7016.019376] LNet: Added LNI 192.168.201.109@tcp116 [8/256/0/180] [ 7016.751279] LNet: Added LNI 192.168.201.109@tcp117 [8/256/0/180] [ 7017.372946] LNet: Added LNI 192.168.201.109@tcp118 [8/256/0/180] [ 7018.089856] LNet: Added LNI 192.168.201.109@tcp119 [8/256/0/180] [ 7018.666861] LNet: Added LNI 192.168.201.109@tcp120 [8/256/0/180] [ 7019.243901] LNet: Added LNI 192.168.201.109@tcp121 [8/256/0/180] [ 7019.863532] LNet: Added LNI 192.168.201.109@tcp122 [8/256/0/180] [ 7020.472234] LNet: Added LNI 192.168.201.109@tcp123 [8/256/0/180] [ 7021.051700] LNet: Added LNI 192.168.201.109@tcp124 [8/256/0/180] [ 7021.590290] LNet: Added LNI 192.168.201.109@tcp125 [8/256/0/180] [ 7022.108712] LNet: Added LNI 192.168.201.109@tcp126 [8/256/0/180] [ 7022.631505] LNet: Added LNI 192.168.201.109@tcp127 [8/256/0/180] [ 7023.187548] LNet: Added LNI 192.168.201.109@tcp128 [8/256/0/180] [ 7023.772396] LNet: Added LNI 192.168.201.109@tcp129 [8/256/0/180] [ 7024.401170] LNet: Added LNI 192.168.201.109@tcp130 [8/256/0/180] [ 7024.952901] LNet: Added LNI 192.168.201.109@tcp131 [8/256/0/180] [ 7025.519864] LNet: Added LNI 192.168.201.109@tcp132 [8/256/0/180] [ 7026.157354] LNet: Added LNI 192.168.201.109@tcp133 [8/256/0/180] [ 7026.769390] LNet: Added LNI 192.168.201.109@tcp134 [8/256/0/180] [ 7027.366787] LNet: Added LNI 192.168.201.109@tcp135 [8/256/0/180] [ 7032.587071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7032.604257] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7033.715152] Lustre: 463507:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [ 7033.773801] LNetError: 460970:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [ 7033.784366] LNetError: Protocol error connecting to 192.168.201.109@tcp115 at host 192.168.201.109:988: is it running a compatible version of Lustre? [ 7033.791189] LNetError: Skipped 2 previous similar messages [ 7033.863651] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7033.899783] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7033.995631] Lustre: lustre-MDT0000: new disk, initializing [ 7034.070464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7034.083080] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7034.655216] LNet: 4 local NIs in recovery (showing 4): 192.168.201.109@tcp127, 192.168.201.109@tcp128, 192.168.201.109@tcp116, 192.168.201.109@tcp129 [ 7034.656979] LNetError: 463528:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [ 7034.677235] LNetError: 463528:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 10 previous similar messages [ 7034.685305] LNetError: Protocol error connecting to 192.168.201.109@tcp127 at host 192.168.201.109:988: is it running a compatible version of Lustre? [ 7034.701356] LNetError: Skipped 7 previous similar messages [ 7035.680453] LNetError: 463529:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [ 7035.691120] LNetError: Protocol error connecting to 192.168.201.109@tcp129 at host 192.168.201.109:988: is it running a compatible version of Lustre? [ 7036.856856] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7037.736213] LNetError: 463536:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [ 7037.742656] LNetError: 463536:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7037.747503] LNetError: Protocol error connecting to 192.168.201.109@tcp129 at host 192.168.201.109:988: is it running a compatible version of Lustre? [ 7037.757596] LNetError: Skipped 1 previous similar message [ 7041.826075] LNetError: 463535:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [ 7041.835318] LNetError: 463535:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 1 previous similar message [ 7041.842830] LNetError: Protocol error connecting to 192.168.201.109@tcp129 at host 192.168.201.109:988: is it running a compatible version of Lustre? [ 7041.853571] LNetError: Skipped 1 previous similar message [ 7044.328059] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.427168] Lustre: 463524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7044.505457] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7044.511129] Lustre: Skipped 1 previous similar message [ 7044.563633] Lustre: lustre-MDT0001: new disk, initializing [ 7044.596332] Lustre: 464325:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [ 7044.601635] Lustre: 464325:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7044.690940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7044.725420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7044.747463] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7047.569743] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7050.018342] LNetError: 463532:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [ 7050.026166] LNetError: 463532:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 19 previous similar messages [ 7050.035519] LNetError: Protocol error connecting to 192.168.201.109@tcp129 at host 192.168.201.109:988: is it running a compatible version of Lustre? [ 7050.048063] LNetError: Skipped 19 previous similar messages [ 7051.279667] Lustre: 464765:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [ 7051.288784] Lustre: 464765:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7054.175521] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7057.858447] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7062.177652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7062.382826] Lustre: lustre-OST0000: new disk, initializing [ 7062.385512] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7062.390410] Lustre: 465498:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7062.452191] Lustre: 465503:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [ 7062.456110] Lustre: 465503:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [ 7062.482598] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7066.267046] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7066.401317] LNetError: 460969:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [ 7066.415918] LNetError: 460969:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [ 7066.422285] LNetError: Protocol error connecting to 192.168.201.109@tcp125 at host 192.168.201.109:988: is it running a compatible version of Lustre? [ 7066.432492] LNetError: Skipped 12 previous similar messages [ 7069.728402] Lustre: 466015:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [ 7069.735731] Lustre: 466015:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [ 7069.788793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7069.803091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7069.897330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7073.698296] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7080.428388] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7080.431973] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7080.442476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7080.457212] Lustre: Skipped 1 previous similar message [ 7085.542642] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7090.658289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7090.665854] Lustre: Skipped 2 previous similar messages [ 7093.215290] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7093.360242] Lustre: server umount lustre-OST0000 complete [ 7095.073063] LNet: 4 local NIs in recovery (showing 4): 192.168.201.109@tcp131, 192.168.201.109@tcp133, 192.168.201.109@tcp134, 192.168.201.109@tcp127 [ 7099.169055] LNetError: 463535:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [ 7099.175451] LNetError: 463535:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 23 previous similar messages [ 7099.183827] LNetError: Protocol error connecting to 192.168.201.109@tcp103 at host 192.168.201.109:988: is it running a compatible version of Lustre? [ 7099.193812] LNetError: Skipped 23 previous similar messages [ 7100.897440] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7100.901378] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7100.904427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7100.905056] Lustre: Skipped 2 previous similar messages [ 7100.921719] Lustre: Skipped 1 previous similar message [ 7102.243412] Lustre: server umount lustre-MDT0000 complete [ 7104.574606] LustreError: 463516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941408 with bad export cookie 16882391652753368129 [ 7104.578788] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7104.583335] LustreError: 463516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7104.823177] Lustre: server umount lustre-MDT0001 complete [ 7111.387928] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7112.871613] Key type lgssc unregistered [ 7113.080244] LNet: 467412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7113.084846] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7114.143324] LNet: Removed LNI 192.168.201.109@tcp [ 7114.145306] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7115.167242] LNet: Removed LNI 192.168.201.109@tcp100 [ 7115.169134] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7115.174126] LNet: Removed LNI 192.168.201.109@tcp101 [ 7116.191433] LNet: Removed LNI 192.168.201.109@tcp102 [ 7117.215156] LNet: Removed LNI 192.168.201.109@tcp103 [ 7117.220233] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7117.233445] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 7117.238889] LNet: Removed LNI 192.168.201.109@tcp104 [ 7118.303237] LNet: Removed LNI 192.168.201.109@tcp105 [ 7119.328936] LNet: Removed LNI 192.168.201.109@tcp106 [ 7120.351470] LNet: Removed LNI 192.168.201.109@tcp107 [ 7121.375176] LNet: Removed LNI 192.168.201.109@tcp108 [ 7121.378720] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7121.383341] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [ 7121.387270] LNet: Removed LNI 192.168.201.109@tcp109 [ 7122.399266] LNet: Removed LNI 192.168.201.109@tcp110 [ 7123.423158] LNet: Removed LNI 192.168.201.109@tcp111 [ 7124.447345] LNet: Removed LNI 192.168.201.109@tcp112 [ 7125.471293] LNet: Removed LNI 192.168.201.109@tcp113 [ 7126.495149] LNet: Removed LNI 192.168.201.109@tcp114 [ 7127.519142] LNet: Removed LNI 192.168.201.109@tcp115 [ 7128.543158] LNet: Removed LNI 192.168.201.109@tcp116 [ 7129.567178] LNet: Removed LNI 192.168.201.109@tcp117 [ 7129.570175] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7129.574334] LNetError: 467412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [ 7129.577932] LNet: Removed LNI 192.168.201.109@tcp118 [ 7130.592313] LNet: Removed LNI 192.168.201.109@tcp119 [ 7131.615489] LNet: Removed LNI 192.168.201.109@tcp120 [ 7132.640660] LNet: Removed LNI 192.168.201.109@tcp121 [ 7133.663141] LNet: Removed LNI 192.168.201.109@tcp122 [ 7134.687141] LNet: Removed LNI 192.168.201.109@tcp123 [ 7135.712594] LNet: Removed LNI 192.168.201.109@tcp124 [ 7136.735138] LNet: Removed LNI 192.168.201.109@tcp125 [ 7137.759562] LNet: Removed LNI 192.168.201.109@tcp126 [ 7137.765323] LNet: Removed LNI 192.168.201.109@tcp127 [ 7138.783143] LNet: Removed LNI 192.168.201.109@tcp128 [ 7139.808253] LNet: Removed LNI 192.168.201.109@tcp129 [ 7140.831348] LNet: Removed LNI 192.168.201.109@tcp130 [ 7140.835771] LNet: Removed LNI 192.168.201.109@tcp131 [ 7141.855271] LNet: Removed LNI 192.168.201.109@tcp132 [ 7142.879112] LNet: Removed LNI 192.168.201.109@tcp133 [ 7142.881212] LNet: Removed LNI 192.168.201.109@tcp134 [ 7143.912860] LNet: Removed LNI 192.168.201.109@tcp135 [ 7144.452835] Key type .llcrypt unregistered [ 7144.454418] Key type ._llcrypt unregistered [ 7159.700398] Key type ._llcrypt registered [ 7159.708630] Key type .llcrypt registered [ 7159.797239] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 7169.375645] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7170.043185] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7170.107601] alg: No test for adler32 (adler32-zlib) [ 7171.059578] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 7171.223300] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7172.863210] Key type lgssc registered [ 7173.591378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7193.580944] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 00:38:15 (1786941495) [ 7203.544602] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7205.164349] Key type lgssc unregistered [ 7205.347897] LNet: 471480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7205.354349] LNetError: 471480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7205.367198] LNet: Removed LNI 192.168.201.109@tcp [ 7205.849643] Key type .llcrypt unregistered [ 7205.851349] Key type ._llcrypt unregistered [ 7223.026195] Key type ._llcrypt registered [ 7223.027985] Key type .llcrypt registered [ 7223.206188] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7224.172815] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7224.388972] alg: No test for adler32 (adler32-zlib) [ 7225.621828] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 7226.227491] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7227.991161] Key type lgssc registered [ 7229.619339] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7240.516697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7240.527741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7242.101965] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7242.173433] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7242.473134] Lustre: lustre-MDT0000: new disk, initializing [ 7242.656486] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7242.766032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7248.593903] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7263.149590] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7263.601600] Lustre: 472694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7263.778376] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7263.781865] Lustre: Skipped 1 previous similar message [ 7264.083621] Lustre: lustre-MDT0001: new disk, initializing [ 7264.330189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7264.519201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7264.536949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7272.174565] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7287.817727] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7295.537760] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7304.092238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7304.584995] Lustre: lustre-OST0000: new disk, initializing [ 7304.589447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7304.594953] Lustre: 474662:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7304.740308] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7310.462351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7310.470100] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7310.648382] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7311.709191] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7321.597340] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7325.270538] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [ 7331.008920] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 7331.036444] LNet: Added LNI 192.168.201.109@tcp10 [8/256/0/180] [ 7338.561440] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 7338.597504] LNet: Added LNI 192.168.201.109@tcp11 [8/256/0/180] [ 7344.233721] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 7344.313459] LNet: Added LNI 192.168.201.109@tcp12 [8/256/0/180] [ 7350.337399] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 7350.396112] LNet: Added LNI 192.168.201.109@tcp13 [8/256/0/180] [ 7355.814373] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 7355.861083] LNet: Added LNI 192.168.201.109@tcp14 [8/256/0/180] [ 7361.646747] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 7361.676588] LNet: Added LNI 192.168.201.109@tcp15 [8/256/0/180] [ 7365.663708] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 7365.690651] LNet: Added LNI 192.168.201.109@tcp16 [8/256/0/180] [ 7370.948635] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 7370.984861] LNet: Added LNI 192.168.201.109@tcp17 [8/256/0/180] [ 7376.216788] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 7376.240698] LNet: Added LNI 192.168.201.109@tcp18 [8/256/0/180] [ 7382.232448] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 7382.251444] LNet: Added LNI 192.168.201.109@tcp19 [8/256/0/180] [ 7386.904472] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 7386.944758] LNet: Added LNI 192.168.201.109@tcp20 [8/256/0/180] [ 7391.583734] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 7391.602496] LNet: Added LNI 192.168.201.109@tcp21 [8/256/0/180] [ 7396.670261] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 7396.690839] LNet: Added LNI 192.168.201.109@tcp22 [8/256/0/180] [ 7401.788813] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 7401.838364] LNet: Added LNI 192.168.201.109@tcp23 [8/256/0/180] [ 7408.165300] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 7408.198700] LNet: Added LNI 192.168.201.109@tcp24 [8/256/0/180] [ 7415.743873] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 7415.830707] LNet: Added LNI 192.168.201.109@tcp25 [8/256/0/180] [ 7422.396961] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 7422.426574] LNet: Added LNI 192.168.201.109@tcp26 [8/256/0/180] [ 7430.901629] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 7430.965257] LNet: Added LNI 192.168.201.109@tcp27 [8/256/0/180] [ 7438.457192] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 7438.493617] LNet: Added LNI 192.168.201.109@tcp28 [8/256/0/180] [ 7444.985786] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 7445.014365] LNet: Added LNI 192.168.201.109@tcp29 [8/256/0/180] [ 7451.722572] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 7451.770202] LNet: Added LNI 192.168.201.109@tcp30 [8/256/0/180] [ 7459.111463] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 7459.157029] LNet: Added LNI 192.168.201.109@tcp31 [8/256/0/180] [ 7466.231276] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 7466.264947] LNet: Added LNI 192.168.201.109@tcp32 [8/256/0/180] [ 7473.359456] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 7473.389642] LNet: Added LNI 192.168.201.109@tcp33 [8/256/0/180] [ 7479.965892] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 7480.004842] LNet: Added LNI 192.168.201.109@tcp34 [8/256/0/180] [ 7487.011720] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 7487.054849] LNet: Added LNI 192.168.201.109@tcp35 [8/256/0/180] [ 7493.608461] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 7493.634151] LNet: Added LNI 192.168.201.109@tcp36 [8/256/0/180] [ 7503.586112] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 7503.639829] LNet: Added LNI 192.168.201.109@tcp37 [8/256/0/180] [ 7511.703327] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 7511.755941] LNet: Added LNI 192.168.201.109@tcp38 [8/256/0/180] [ 7519.459799] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 7519.487928] LNet: Added LNI 192.168.201.109@tcp39 [8/256/0/180] [ 7527.504569] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 7527.559749] LNet: Added LNI 192.168.201.109@tcp40 [8/256/0/180] [ 7536.753977] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 7536.802770] LNet: Added LNI 192.168.201.109@tcp41 [8/256/0/180] [ 7545.679123] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 7545.697423] LNet: Added LNI 192.168.201.109@tcp42 [8/256/0/180] [ 7553.187196] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 7553.223644] LNet: Added LNI 192.168.201.109@tcp43 [8/256/0/180] [ 7561.473461] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 7561.494789] LNet: Added LNI 192.168.201.109@tcp44 [8/256/0/180] [ 7570.464490] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 7570.523090] LNet: Added LNI 192.168.201.109@tcp45 [8/256/0/180] [ 7579.008126] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 7579.071243] LNet: Added LNI 192.168.201.109@tcp46 [8/256/0/180] [ 7587.932606] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 7587.978369] LNet: Added LNI 192.168.201.109@tcp47 [8/256/0/180] [ 7597.826242] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 7597.901153] LNet: Added LNI 192.168.201.109@tcp48 [8/256/0/180] [ 7606.780379] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 7606.821867] LNet: Added LNI 192.168.201.109@tcp49 [8/256/0/180] [ 7616.890174] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 7616.921099] LNet: Added LNI 192.168.201.109@tcp50 [8/256/0/180] [ 7643.106773] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7643.110483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7643.143850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7643.617730] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7646.726677] Lustre: server umount lustre-MDT0000 complete [ 7651.213743] LustreError: 472683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786941954 with bad export cookie 2081332295564555997 [ 7651.217406] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7651.231241] LustreError: 472683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7651.691931] Lustre: server umount lustre-MDT0001 complete [ 7670.239103] Lustre: 471964:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786941957/real 1786941957] req@0000000019213d2b x1873744000886656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786941973 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7670.267025] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7670.276023] Lustre: Skipped 1 previous similar message [ 7673.311401] Lustre: 471966:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786941960/real 1786941960] req@000000001808c078 x1873744000886784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786941976 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7673.554508] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7675.439337] Lustre: 471966:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786941962/real 1786941962] req@0000000099fa88a2 x1873744000886912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786941978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7681.503459] Lustre: 471964:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1786941969/real 1786941969] req@00000000182467f4 x1873744000887296/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1786941985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7681.537848] Lustre: 471964:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7683.743254] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7696.308260] LustreError: 474655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7696.325360] LustreError: 474655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7696.418350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7699.426205] LustreError: 474656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7700.450572] LustreError: 474655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7700.457636] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7702.177440] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7705.576212] LustreError: 474656:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7710.695438] LustreError: 474655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7711.689422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7712.118699] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7715.190873] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 7717.651280] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7726.715427] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7732.720481] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7752.674036] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7752.682236] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7752.692751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7752.697935] Lustre: Skipped 2 previous similar messages [ 7756.274422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7756.290220] Lustre: Skipped 2 previous similar messages [ 7758.966752] Lustre: server umount lustre-OST0000 complete [ 7766.499733] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7766.507694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7766.516593] Lustre: Skipped 2 previous similar messages [ 7766.552337] Lustre: Skipped 1 previous similar message [ 7771.183845] Lustre: server umount lustre-MDT0000 complete [ 7771.620820] LustreError: 498932:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7771.642364] LustreError: 498932:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7775.643982] LustreError: 473487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942079 with bad export cookie 2081332295564558601 [ 7775.646740] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7775.655537] LustreError: 473487:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7776.055565] Lustre: server umount lustre-MDT0001 complete [ 7788.200623] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7790.928286] Key type lgssc unregistered [ 7791.312134] LNet: 501738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7791.322050] LNetError: 501738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7791.328463] LNet: Removed LNI 192.168.201.109@tcp [ 7791.331640] LNet: Removed LNI 192.168.201.109@tcp10 [ 7791.333564] LNet: Removed LNI 192.168.201.109@tcp11 [ 7791.337385] LNet: Removed LNI 192.168.201.109@tcp12 [ 7791.339762] LNet: Removed LNI 192.168.201.109@tcp13 [ 7791.342523] LNet: Removed LNI 192.168.201.109@tcp14 [ 7791.345727] LNet: Removed LNI 192.168.201.109@tcp15 [ 7791.348678] LNet: Removed LNI 192.168.201.109@tcp16 [ 7791.353550] LNet: Removed LNI 192.168.201.109@tcp17 [ 7791.355480] LNet: Removed LNI 192.168.201.109@tcp18 [ 7791.357728] LNet: Removed LNI 192.168.201.109@tcp19 [ 7791.359790] LNet: Removed LNI 192.168.201.109@tcp20 [ 7791.361928] LNet: Removed LNI 192.168.201.109@tcp21 [ 7791.364194] LNet: Removed LNI 192.168.201.109@tcp22 [ 7791.366218] LNet: Removed LNI 192.168.201.109@tcp23 [ 7791.368416] LNet: Removed LNI 192.168.201.109@tcp24 [ 7791.370459] LNet: Removed LNI 192.168.201.109@tcp25 [ 7791.372876] LNet: Removed LNI 192.168.201.109@tcp26 [ 7791.375255] LNet: Removed LNI 192.168.201.109@tcp27 [ 7791.377452] LNet: Removed LNI 192.168.201.109@tcp28 [ 7791.380200] LNet: Removed LNI 192.168.201.109@tcp29 [ 7791.382282] LNet: Removed LNI 192.168.201.109@tcp30 [ 7791.384582] LNet: Removed LNI 192.168.201.109@tcp31 [ 7791.386945] LNet: Removed LNI 192.168.201.109@tcp32 [ 7791.389590] LNet: Removed LNI 192.168.201.109@tcp33 [ 7791.391792] LNet: Removed LNI 192.168.201.109@tcp34 [ 7791.394308] LNet: Removed LNI 192.168.201.109@tcp35 [ 7791.396730] LNet: Removed LNI 192.168.201.109@tcp36 [ 7791.399231] LNet: Removed LNI 192.168.201.109@tcp37 [ 7791.401316] LNet: Removed LNI 192.168.201.109@tcp38 [ 7791.403927] LNet: Removed LNI 192.168.201.109@tcp39 [ 7791.406221] LNet: Removed LNI 192.168.201.109@tcp40 [ 7791.408243] LNet: Removed LNI 192.168.201.109@tcp41 [ 7791.410637] LNet: Removed LNI 192.168.201.109@tcp42 [ 7791.412896] LNet: Removed LNI 192.168.201.109@tcp43 [ 7791.414947] LNet: Removed LNI 192.168.201.109@tcp44 [ 7791.417443] LNet: Removed LNI 192.168.201.109@tcp45 [ 7791.419749] LNet: Removed LNI 192.168.201.109@tcp46 [ 7791.421542] LNet: Removed LNI 192.168.201.109@tcp47 [ 7791.423870] LNet: Removed LNI 192.168.201.109@tcp48 [ 7791.426243] LNet: Removed LNI 192.168.201.109@tcp49 [ 7791.435980] LNet: Removed LNI 192.168.201.109@tcp50 [ 7792.088995] Key type .llcrypt unregistered [ 7792.090420] Key type ._llcrypt unregistered [ 7812.922761] Key type ._llcrypt registered [ 7812.924663] Key type .llcrypt registered [ 7813.020314] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 7828.939273] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7830.347436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7830.425057] alg: No test for adler32 (adler32-zlib) [ 7831.446855] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 7831.659528] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7833.399881] Key type lgssc registered [ 7834.296930] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7867.906759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7867.984515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7869.299742] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7869.345553] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7869.481101] Lustre: lustre-MDT0000: new disk, initializing [ 7869.632648] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7869.666558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7874.153275] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7885.739568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7885.910824] Lustre: 504936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7885.962923] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7885.966631] Lustre: Skipped 1 previous similar message [ 7886.027977] Lustre: lustre-MDT0001: new disk, initializing [ 7886.123705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7886.155206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7886.168582] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7891.068682] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7901.398485] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7906.854991] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7914.656917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7915.020662] Lustre: *** cfs_fail_loc=90f, val=0*** [ 7915.120779] Lustre: lustre-OST0000: new disk, initializing [ 7915.124644] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7915.134681] Lustre: 506953:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7915.270942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7920.681420] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7920.696492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7920.832340] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7921.857922] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7931.272982] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7941.604434] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7941.607145] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7941.617743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7941.619896] Lustre: Skipped 1 previous similar message [ 7946.068353] Lustre: server umount lustre-OST0000 complete [ 7951.841417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7951.845283] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7951.860205] Lustre: Skipped 1 previous similar message [ 7951.869125] Lustre: Skipped 2 previous similar messages [ 7956.529347] Lustre: server umount lustre-MDT0000 complete [ 7960.086414] LustreError: 504927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942263 with bad export cookie 18273876859363194303 [ 7960.087422] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.096897] LustreError: 504927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7960.530088] Lustre: server umount lustre-MDT0001 complete [ 7972.312825] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7974.703691] Key type lgssc unregistered [ 7974.951927] LNet: 508861:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7974.967464] LNetError: 508861:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7974.986870] LNet: Removed LNI 192.168.201.109@tcp [ 7975.889239] Key type .llcrypt unregistered [ 7975.899346] Key type ._llcrypt unregistered [ 7993.614296] Key type ._llcrypt registered [ 7993.616101] Key type .llcrypt registered [ 7993.728576] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7994.361141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7994.471612] alg: No test for adler32 (adler32-zlib) [ 7995.452553] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 7995.803413] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7997.543388] Key type lgssc registered [ 7998.529075] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8005.405984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8005.421492] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8006.780823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8011.019529] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8017.383038] LustreError: 510136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8017.415203] LustreError: 510136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8018.706516] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8019.045132] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8023.408197] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8030.585854] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8036.311142] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8041.883734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8042.237752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8047.602264] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8055.215447] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8066.041903] loop0: detected capacity change from 0 to 204800000 [ 8066.256981] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8066.274745] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8066.295046] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8072.229813] loop0: detected capacity change from 0 to 204800000 [ 8072.449241] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8072.462166] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8072.475836] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8077.475637] loop0: detected capacity change from 0 to 204800000 [ 8083.310970] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8083.512611] Lustre: fs15246-OST0000: new disk, initializing [ 8083.520714] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8083.535469] Lustre: 513729:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8083.621901] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8089.679489] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8098.913934] loop1: detected capacity change from 0 to 204800000 [ 8104.432899] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8104.553699] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8104.605152] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8104.707537] Lustre: fs15246-MDT0000: new disk, initializing [ 8104.863236] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.897290] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8105.017857] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8106.080801] Lustre: 514852:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000a2f09081 x1873744808163200/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1786942425 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8108.747855] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8111.148492] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8111.158330] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8111.199181] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8131.048166] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8131.078074] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [ 8136.187690] LustreError: 510132:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8136.342836] Lustre: server umount fs15246-MDT0000 complete [ 8141.306341] LustreError: 510132:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8144.947773] Lustre: server umount fs15246-OST0000 complete [ 8155.105612] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8155.112156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8155.128429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8156.641409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8156.651263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8160.797283] Lustre: server umount lustre-OST0000 complete [ 8166.887728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8166.893326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8166.900638] Lustre: Skipped 1 previous similar message [ 8166.909069] Lustre: Skipped 2 previous similar messages [ 8170.484880] Lustre: server umount lustre-MDT0000 complete [ 8172.008823] LustreError: 510136:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.031026] LustreError: 510136:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8173.636705] LustreError: 514846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942477 with bad export cookie 15249625070732615967 [ 8173.642739] LustreError: 514846:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8173.643147] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8173.995469] Lustre: server umount lustre-MDT0001 complete [ 8183.160883] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8184.974719] Key type lgssc unregistered [ 8185.186723] LNet: 517948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8185.192685] LNetError: 517948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8185.204785] LNet: Removed LNI 192.168.201.109@tcp [ 8185.911122] Key type .llcrypt unregistered [ 8185.918321] Key type ._llcrypt unregistered [ 8199.291190] Key type ._llcrypt registered [ 8199.292845] Key type .llcrypt registered [ 8199.382283] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8200.064599] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8200.122275] alg: No test for adler32 (adler32-zlib) [ 8201.198247] Lustre: Lustre: Build Version: 2.17.57_1_g37263e8 [ 8201.411369] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8203.087307] Key type lgssc registered [ 8204.214825] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8236.985411] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 8244.587501] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8273.453264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8273.473141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8274.738604] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8274.771561] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8274.839684] Lustre: lustre-MDT0000: new disk, initializing [ 8274.892323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.907168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8278.416984] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8287.899817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8288.037626] Lustre: 522390:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8288.099905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8288.102933] Lustre: Skipped 1 previous similar message [ 8288.159303] Lustre: lustre-MDT0001: new disk, initializing [ 8288.259173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8288.293128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8288.303216] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8292.354579] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8301.875850] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8306.545548] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8311.959441] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8312.167406] Lustre: lustre-OST0000: new disk, initializing [ 8312.170841] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8312.175765] Lustre: 524358:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8312.253955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8316.569061] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8319.037849] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8319.048894] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8319.102591] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8324.012949] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8328.506709] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8328.791064] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8332.840622] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8333.035527] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8334.818049] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8334.825095] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8334.834344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8334.846682] Lustre: Skipped 1 previous similar message [ 8334.864224] Lustre: Skipped 1 previous similar message [ 8339.939364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8339.944165] Lustre: Skipped 1 previous similar message [ 8340.661872] Lustre: server umount lustre-OST0000 complete [ 8344.545925] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8344.554121] LustreError: Skipped 1 previous similar message [ 8344.555886] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8344.565934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8349.930488] Lustre: server umount lustre-MDT0000 complete [ 8353.999874] LustreError: 522382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1786942657 with bad export cookie 14828281843304102116 [ 8354.009780] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8354.363342] Lustre: server umount lustre-MDT0001 complete [ 8377.299146] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 8386.613777] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8409.753913] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8275 sec ========== 00:58:31 (1786942711) [ 8411.155830] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:58:32 (1786942712) === [ 8413.696796] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:58:35 (1786942715) === [ 8443.136259] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8444.986911] Key type lgssc unregistered [ 8445.171496] LNet: 531000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8445.175607] LNetError: 531000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8445.183208] LNet: Removed LNI 192.168.201.109@tcp [ 8445.720598] Key type .llcrypt unregistered [ 8445.722429] Key type ._llcrypt unregistered