[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 494077877 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003306] x2apic enabled [ 0.004009] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008582] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011125] LSM: Security Framework initializing [ 0.012048] Yama: becoming mindful. [ 0.013035] SELinux: Initializing. [ 0.015016] *** VALIDATE selinux *** [ 0.022710] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026773] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027134] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028096] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] *** VALIDATE tmpfs *** [ 0.031296] *** VALIDATE proc *** [ 0.032229] *** VALIDATE cgroup *** [ 0.033008] *** VALIDATE cgroup2 *** [ 0.034264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035142] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037026] Spectre V2 : User space: Vulnerable [ 0.038009] Speculative Store Bypass: Vulnerable [ 0.041437] debug: unmapping init [mem 0xffffffffa3259000-0xffffffffa3260fff] [ 0.044142] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045685] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046023] ... version: 2 [ 0.047010] ... bit width: 48 [ 0.048012] ... generic registers: 4 [ 0.049011] ... value mask: 0000ffffffffffff [ 0.050014] ... max period: 00007fffffffffff [ 0.051013] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.054235] rcu: Hierarchical SRCU implementation. [ 0.056167] smp: Bringing up secondary CPUs ... [ 0.057551] x86: Booting SMP configuration: [ 0.058029] .... node #0, CPUs: #1 #2 #3 [ 0.061566] smp: Brought up 1 node, 4 CPUs [ 0.063029] smpboot: Max logical packages: 1 [ 0.064019] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.204083] node 0 deferred pages initialised in 138ms [ 0.208297] devtmpfs: initialized [ 0.209215] x86/mm: Memory block size: 128MB [ 0.211749] gcov: version magic: 0x41383552 [ 0.212583] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.216126] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.219442] pinctrl core: initialized pinctrl subsystem [ 0.221247] [ 0.222013] ************************************************************* [ 0.225019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.228021] ** ** [ 0.230016] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.233014] ** ** [ 0.235018] ** This means that this kernel is built to expose internal ** [ 0.238017] ** IOMMU data structures, which may compromise security on ** [ 0.241019] ** your system. ** [ 0.242000] ** ** [ 0.244011] ** If you see this message and you are not debugging the ** [ 0.246015] ** kernel, report this immediately to your vendor! ** [ 0.248018] ** ** [ 0.251020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.254017] ************************************************************* [ 0.256751] NET: Registered protocol family 16 [ 0.259509] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.260090] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.264104] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.268114] cpuidle: using governor menu [ 0.270727] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.273675] PCI: Using configuration type 1 for base access [ 0.275120] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.285110] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.288138] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.292040] cryptd: max_cpu_qlen set to 1000 [ 0.295340] ACPI: Added _OSI(Module Device) [ 0.296014] ACPI: Added _OSI(Processor Device) [ 0.297008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.298009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.302910] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.311545] ACPI: Interpreter enabled [ 0.312074] ACPI: PM: (supports S0 S3 S4 S5) [ 0.313012] ACPI: Using IOAPIC for interrupt routing [ 0.314142] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.317664] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.328301] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.330040] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.333018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.336077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.341446] acpiphp: Slot [2] registered [ 0.343139] acpiphp: Slot [5] registered [ 0.345210] acpiphp: Slot [6] registered [ 0.346155] acpiphp: Slot [7] registered [ 0.348156] acpiphp: Slot [8] registered [ 0.349174] acpiphp: Slot [9] registered [ 0.350318] acpiphp: Slot [10] registered [ 0.351000] acpiphp: Slot [3] registered [ 0.351000] acpiphp: Slot [4] registered [ 0.354152] acpiphp: Slot [11] registered [ 0.355154] acpiphp: Slot [12] registered [ 0.357213] acpiphp: Slot [13] registered [ 0.359148] acpiphp: Slot [14] registered [ 0.360173] acpiphp: Slot [15] registered [ 0.362119] acpiphp: Slot [16] registered [ 0.363115] acpiphp: Slot [17] registered [ 0.364087] acpiphp: Slot [18] registered [ 0.365103] acpiphp: Slot [19] registered [ 0.366094] acpiphp: Slot [20] registered [ 0.368121] acpiphp: Slot [21] registered [ 0.369125] acpiphp: Slot [22] registered [ 0.371100] acpiphp: Slot [23] registered [ 0.372103] acpiphp: Slot [24] registered [ 0.374098] acpiphp: Slot [25] registered [ 0.375098] acpiphp: Slot [26] registered [ 0.376100] acpiphp: Slot [27] registered [ 0.377158] acpiphp: Slot [28] registered [ 0.379171] acpiphp: Slot [29] registered [ 0.381142] acpiphp: Slot [30] registered [ 0.383132] acpiphp: Slot [31] registered [ 0.384106] PCI host bridge to bus 0000:00 [ 0.386034] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.389022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.392042] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.395023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.397018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.399026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.401180] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.404011] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.407360] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.416761] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.423011] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.427028] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.430025] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.434022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.436519] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.439973] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.444061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.447925] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.454021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.468022] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.473021] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.479393] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.488022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.497024] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.515021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.524797] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.540027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.558021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.581017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.595218] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.605021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.612024] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.631023] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.646020] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.653023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.660021] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.676089] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.685612] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.693013] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.701016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.717016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.727098] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.735032] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.743019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.762000] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.775906] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.778414] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.781360] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.783345] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.786255] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.791189] iommu: Default domain type: Passthrough [ 0.792901] SCSI subsystem initialized [ 0.796290] ACPI: bus type USB registered [ 0.797184] usbcore: registered new interface driver usbfs [ 0.800141] usbcore: registered new interface driver hub [ 0.802115] usbcore: registered new device driver usb [ 0.804230] pps_core: LinuxPPS API ver. 1 registered [ 0.806017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.809098] PTP clock support registered [ 0.812051] EDAC MC: Ver: 3.0.0 [ 0.814177] PCI: Using ACPI for IRQ routing [ 0.815996] NetLabel: Initializing [ 0.816012] NetLabel: domain hash size = 128 [ 0.817020] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.818129] NetLabel: unlabeled traffic allowed by default [ 0.821057] vgaarb: loaded [ 0.822195] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.824017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.831454] clocksource: Switched to clocksource kvm-clock [ 0.943884] VFS: Disk quotas dquot_6.6.0 [ 0.946322] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.948751] *** VALIDATE ramfs *** [ 0.949957] *** VALIDATE hugetlbfs *** [ 0.952286] pnp: PnP ACPI init [ 0.955192] pnp: PnP ACPI: found 6 devices [ 0.975663] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.978880] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.980951] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.983341] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.987694] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.989965] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.993409] NET: Registered protocol family 2 [ 0.996214] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.001336] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.005520] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.012280] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.015892] TCP: Hash tables configured (established 65536 bind 65536) [ 1.020542] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.025266] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.028358] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.031891] NET: Registered protocol family 1 [ 1.035248] RPC: Registered named UNIX socket transport module. [ 1.037693] RPC: Registered udp transport module. [ 1.039802] RPC: Registered tcp transport module. [ 1.041835] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.044587] NET: Registered protocol family 44 [ 1.046757] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.048930] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.051560] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.054617] PCI: CLS 0 bytes, default 64 [ 1.058080] Unpacking initramfs... [ 2.522895] debug: unmapping init [mem 0xffff99ad3cc54000-0xffff99ad3ffbffff] [ 2.530233] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.532386] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.536181] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.053027] Initialise system trusted keyrings [ 3.055099] Key type blacklist registered [ 3.057398] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.066592] zbud: loaded [ 3.070165] *** VALIDATE nfs *** [ 3.071513] *** VALIDATE nfs4 *** [ 3.073239] pstore: using deflate compression [ 3.076782] Platform Keyring initialized [ 3.188558] NET: Registered protocol family 38 [ 3.190456] Key type asymmetric registered [ 3.191536] Asymmetric key parser 'x509' registered [ 3.193976] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.196594] io scheduler mq-deadline registered [ 3.198175] io scheduler kyber registered [ 3.199556] io scheduler bfq registered [ 3.200884] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.203860] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.207040] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.210498] ACPI: Power Button [PWRF] [ 3.215638] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.223292] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.239739] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.248951] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.269695] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.298298] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.328092] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.333437] Non-volatile memory driver v1.3 [ 3.336473] Linux agpgart interface v0.103 [ 3.370836] virtio_blk virtio1: [vda] 146672 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.374044] vda: detected capacity change from 0 to 75096064 [ 3.394593] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.397743] vdb: detected capacity change from 0 to 1073741824 [ 3.414545] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.419739] vdc: detected capacity change from 0 to 2621440000 [ 3.437491] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.440830] vdd: detected capacity change from 0 to 2621440000 [ 3.463195] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.466477] vde: detected capacity change from 0 to 4294967296 [ 3.490332] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.493637] vdf: detected capacity change from 0 to 4294967296 [ 3.502562] libphy: Fixed MDIO Bus: probed [ 3.508455] usbcore: registered new interface driver usbserial_generic [ 3.510978] usbserial: USB Serial support registered for generic [ 3.515816] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.520429] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.522654] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.525241] mousedev: PS/2 mouse device common for all mice [ 3.528491] rtc_cmos 00:05: RTC can wake from S4 [ 3.531821] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.536173] rtc_cmos 00:05: registered as rtc0 [ 3.538580] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.542293] intel_pstate: CPU model not supported [ 3.546524] hid: raw HID events driver (C) Jiri Kosina [ 3.546919] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.551722] usbcore: registered new interface driver usbhid [ 3.558501] usbhid: USB HID core driver [ 3.560176] drop_monitor: Initializing network drop monitor service [ 3.562814] Initializing XFRM netlink socket [ 3.564862] NET: Registered protocol family 10 [ 3.568360] Segment Routing with IPv6 [ 3.568488] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.569568] NET: Registered protocol family 17 [ 3.573683] mpls_gso: MPLS GSO support [ 3.579701] RAS: Correctable Errors collector initialized. [ 3.581024] AVX version of gcm_enc/dec engaged. [ 3.582047] AES CTR mode by8 optimization enabled [ 3.668781] sched_clock: Marking stable (3668733791, 0)->(4581988245, -913254454) [ 3.671769] registered taskstats version 1 [ 3.673690] Loading compiled-in X.509 certificates [ 3.675635] zswap: loaded using pool lzo/zbud [ 3.702609] Key type big_key registered [ 3.717226] Key type encrypted registered [ 3.718917] ima: No TPM chip found, activating TPM-bypass! [ 3.720768] ima: Allocated hash algorithm: sha1 [ 3.722574] ima: No architecture policies found [ 3.724403] evm: Initialising EVM extended attributes: [ 3.726439] evm: security.selinux [ 3.727783] evm: security.ima [ 3.728941] evm: security.capability [ 3.730617] evm: HMAC attrs: 0x1 [ 3.733298] rtc_cmos 00:05: setting system clock to 2026-08-27 14:22:26 UTC (1787840546) [ 3.742141] debug: unmapping init [mem 0xffffffffa4203000-0xffffffffa43fffff] [ 3.745577] debug: unmapping init [mem 0xffffffffa2f82000-0xffffffffa3258fff] [ 3.754354] Write protecting the kernel read-only data: 28672k [ 3.758084] debug: unmapping init [mem 0xffffffffa1603000-0xffffffffa17fffff] [ 3.760947] debug: unmapping init [mem 0xffffffffa1f14000-0xffffffffa1ffffff] [ 3.798130] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.807353] systemd[1]: Detected virtualization kvm. [ 3.812091] systemd[1]: Detected architecture x86-64. [ 3.813716] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.839745] systemd[1]: No hostname configured. [ 3.842033] systemd[1]: Set hostname to . [ 3.844755] random: systemd: uninitialized urandom read (16 bytes read) [ 3.849410] systemd[1]: Initializing machine ID from random generator. [ 3.908823] random: ln: uninitialized urandom read (6 bytes read) [ 4.003934] random: systemd: uninitialized urandom read (16 bytes read) [ 4.006656] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.011274] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.015830] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Timers. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.736127] device-mapper: uevent: version 1.0.3 [ 4.738145] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.533358] scsi host0: ata_piix [ 5.535992] scsi host1: ata_piix [ 5.537726] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.542816] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 5.547249] virtio_net virtio0 ens2: renamed from eth0 [ 5.551479] random: fast init done [ 9.444647] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.236610] random: crng init done [ 10.238493] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.889834] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.170090] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.439643] SELinux: Disabled at runtime. [ 12.501915] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.511702] systemd[1]: Detected virtualization kvm. [ 12.513791] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 13.042421] systemd[1]: initrd-switch-root.service: Succeeded. [ 13.046280] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 13.052803] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 13.056877] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 13.060819] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 13.069474] systemd[1]: Starting Journal Service... Starting Journal Service... [ 13.079848] systemd[1]: Mounting Kernel Debug File System... Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. [ OK ] Stopped target Switch Root. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Activating swap /dev/disk/by-label/SWAP... [ OK ] Reached target rpc_pipefs.target. Mounting Huge Pages File System... [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Stopped target Initrd File Systems. [ 13.149889] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target Slices. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting POSIX Message Queue File System... [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.562348] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.816488] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.884170] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.039057] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.048839] EDAC sbridge: Ver: 1.1.2 [ 15.952292] Key type dns_resolver registered [ 16.270967] NFS: Registering the id_resolver key type [ 16.273854] Key type id_resolver registered [ 16.276029] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg624-server login: [ 48.887782] libcfs: loading out-of-tree module taints kernel. [ 48.924902] Key type ._llcrypt registered [ 48.927753] Key type .llcrypt registered [ 49.073504] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 64.868588] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 66.072802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.096232] alg: No test for adler32 (adler32-zlib) [ 67.581820] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 68.311873] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 70.039630] Key type lgssc registered [ 71.573523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 84.005168] hrtimer: interrupt took 4154869 ns [ 84.545850] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 116.418136] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 127.721227] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 127.748323] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 128.933866] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 128.980863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 129.067331] Lustre: lustre-MDT0000: new disk, initializing [ 129.141580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 129.156231] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 132.537236] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 143.622334] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.731818] Lustre: 6513:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 143.756893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 143.761664] Lustre: Skipped 1 previous similar message [ 143.819685] Lustre: lustre-MDT0001: new disk, initializing [ 143.863138] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 143.885709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 143.905111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 147.034956] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 151.230277] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 158.379923] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 158.582046] Lustre: lustre-OST0000: new disk, initializing [ 158.588275] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 158.597207] Lustre: 8448:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 158.608668] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 158.661849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 163.466670] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 163.825360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 163.838580] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 163.888356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 175.144732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 175.254482] Lustre: lustre-OST0001: new disk, initializing [ 175.263622] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 175.273353] Lustre: 9522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 175.359896] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 180.529594] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 184.428646] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 184.436663] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 184.523529] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 194.122666] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 202.597407] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 209.538604] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing check_logdir /tmp/testlogs/ [ 214.723102] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing yml_node [ 219.076943] Lustre: DEBUG MARKER: Client: 2.17.57.65 [ 221.274628] Lustre: DEBUG MARKER: MDS: 2.17.57.65 [ 223.462039] Lustre: DEBUG MARKER: OSS: 2.17.57.65 [ 224.878254] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 10:26:05 EDT 2026 [ 240.630685] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 242.475758] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 261.602066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 261.602092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 261.606881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 261.630212] Lustre: Skipped 2 previous similar messages [ 263.845883] Lustre: server umount lustre-MDT0000 complete [ 266.779938] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840809 with bad export cookie 17538923013013225733 [ 266.782959] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 266.785962] LustreError: 6502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 271.844283] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 271.845762] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 271.855958] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 271.855967] Lustre: Skipped 3 previous similar messages [ 271.882217] LustreError: 6518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 271.887412] Lustre: Skipped 2 previous similar messages [ 276.959826] LustreError: 7821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 276.965602] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 276.990337] LustreError: 7821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 276.994397] Lustre: Skipped 2 previous similar messages [ 277.984428] LustreError: 7821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 277.994650] LustreError: 7821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 281.055138] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 281.182215] Lustre: server umount lustre-MDT0001 complete [ 284.072269] Lustre: server umount lustre-OST0000 complete [ 286.890468] Lustre: server umount lustre-OST0001 complete [ 291.872990] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 298.597570] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 331.620547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 331.920450] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 331.959809] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 332.066985] Lustre: lustre-MDT0000: new disk, initializing [ 332.172122] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 332.194615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 335.933585] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 344.304337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 344.410120] Lustre: 15072:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 344.461192] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 344.467263] Lustre: Skipped 1 previous similar message [ 344.532678] Lustre: lustre-MDT0001: new disk, initializing [ 344.641649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 344.725440] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 344.742321] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 348.497269] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 356.508916] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 361.718761] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 367.934539] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 368.297954] Lustre: lustre-OST0000: new disk, initializing [ 368.304321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 368.310287] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 368.393488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 369.893091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 369.900512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 370.006232] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 374.740940] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 383.555209] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 387.980748] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 388.167096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 393.305593] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 393.467185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 395.752414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 395.753712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 395.766729] LustreError: Skipped 1 previous similar message [ 395.767847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 395.767855] Lustre: Skipped 2 previous similar messages [ 395.814676] Lustre: Skipped 1 previous similar message [ 401.305878] Lustre: server umount lustre-OST0000 complete [ 405.984803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 405.989837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 406.002438] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 406.004942] Lustre: Skipped 1 previous similar message [ 406.014534] Lustre: Skipped 3 previous similar messages [ 410.891953] Lustre: server umount lustre-MDT0000 complete [ 411.107229] LustreError: 17428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 411.126995] LustreError: 17428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 414.108594] LustreError: 15869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840956 with bad export cookie 17538923013013227259 [ 414.114795] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 414.119303] LustreError: 15869:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 414.362834] Lustre: server umount lustre-MDT0001 complete [ 419.370134] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 10:29:20 (1787840960) [ 441.740531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 442.485387] LustreError: 19918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 442.599803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 447.966851] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 453.104569] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 453.124604] LustreError: 19919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 456.619342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 461.731715] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 468.189597] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 473.066718] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 474.899887] Lustre: Failing over lustre-MDT0000 [ 475.193293] Lustre: server umount lustre-MDT0000 complete [ 477.664761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 477.666335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 477.666494] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 477.675629] Lustre: Skipped 1 previous similar message [ 493.473522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.656192] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 497.844900] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 499.168496] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 499.178742] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 499.207430] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 506.985147] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 513.507094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 513.521289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 513.530208] Lustre: Skipped 2 previous similar messages [ 517.403156] Lustre: server umount lustre-MDT0000 complete [ 518.625465] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 518.671014] LustreError: 19913:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 521.338170] LustreError: 19898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841064 with bad export cookie 17538923013013228680 [ 521.347458] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 521.347463] LustreError: 19898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 521.691460] Lustre: server umount lustre-MDT0001 complete [ 529.178515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.734818] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 529.741536] Lustre: Skipped 2 previous similar messages [ 533.252152] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 541.427671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 547.257728] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 556.125342] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 563.478107] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 566.037368] Lustre: Failing over lustre-MDT0000 [ 566.356264] Lustre: server umount lustre-MDT0000 complete [ 567.079467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 567.112142] Lustre: Skipped 1 previous similar message [ 582.624240] Lustre: 3649:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841109/real 1787841109] req@ffff99adba0dda40 x1874686756701312/t0(0) o400->MGC192.168.206.124@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787841125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 582.636354] LustreError: 23257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 582.661498] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 582.695756] LustreError: 23257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 585.751703] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 593.889582] LustreError: 3645:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff99ac83460b40 x1874686756704256/t0(0) o250->MGC192.168.206.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 598.481336] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 599.527904] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 599.529536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 599.559858] Lustre: Skipped 1 previous similar message [ 599.572813] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 607.664778] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 612.576373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 612.578861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 612.593037] Lustre: Skipped 2 previous similar messages [ 617.864600] Lustre: server umount lustre-MDT0000 complete [ 621.528523] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841164 with bad export cookie 17538923013013229618 [ 621.533895] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 621.540063] LustreError: 23236:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 621.833747] Lustre: server umount lustre-MDT0001 complete [ 630.350980] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 10:32:50 (1787841170) [ 646.486475] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 655.724900] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 698.103281] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 708.671935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 708.849805] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 708.869962] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 708.961118] Lustre: lustre-MDT0000: new disk, initializing [ 709.073105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 709.076106] Lustre: Skipped 2 previous similar messages [ 709.091616] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 713.958171] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 725.475715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.609734] Lustre: 29632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 725.906324] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 730.988826] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 739.864948] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 744.971301] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 751.299861] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 751.685193] Lustre: 31601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 752.931402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 753.000327] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 753.042362] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 753.079160] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 758.626617] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 769.303427] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 775.468921] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 799.220835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 799.222858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 799.224105] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 799.224113] Lustre: Skipped 1 previous similar message [ 799.246425] Lustre: Skipped 1 previous similar message [ 799.254388] LustreError: Skipped 1 previous similar message [ 804.849901] Lustre: server umount lustre-OST0000 complete [ 815.192969] Lustre: server umount lustre-MDT0000 complete [ 819.459764] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841362 with bad export cookie 17538923013013230024 [ 819.463463] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 819.468835] LustreError: 29624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 819.970148] Lustre: server umount lustre-MDT0001 complete [ 829.513244] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 832.122973] Key type lgssc unregistered [ 832.435335] LNet: 33583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.456348] LNetError: 33583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.471709] LNet: Removed LNI 192.168.206.124@tcp [ 833.378295] Key type .llcrypt unregistered [ 833.380845] Key type ._llcrypt unregistered [ 854.861174] Key type ._llcrypt registered [ 854.863516] Key type .llcrypt registered [ 854.959931] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 856.107640] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 856.247290] alg: No test for adler32 (adler32-zlib) [ 857.278168] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 857.480174] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 859.119240] Key type lgssc registered [ 860.029345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.240635] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 867.270564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 868.794636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 873.393127] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 879.072879] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.094882] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 882.122642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 882.559030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 887.129782] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 894.614946] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 900.022973] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 906.413955] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 906.763606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 907.835763] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 912.739563] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 914.008522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 919.894496] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 932.112395] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 10:37:52 (1787841472) [ 933.741670] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 935.426605] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 942.212728] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 10:38:02 (1787841482) [ 950.211847] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 10:38:10 (1787841490) [ 951.248281] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 960.054566] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 960.067385] Lustre: Skipped 2 previous similar messages [ 961.915856] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 10:38:22 (1787841502) [ 964.061793] Lustre: Setting parameter general.osc.*.max_pages_per_rpc=1024 in log params [ 964.071269] Lustre: Skipped 2 previous similar messages [ 972.794894] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 972.806455] Lustre: Skipped 4 previous similar messages [ 988.062412] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 10:38:48 (1787841528) [ 991.739435] Lustre: *** cfs_fail_loc=131b, val=2*** [ 996.692654] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1007.192238] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 10:39:07 (1787841547) [ 1009.069695] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1009.076682] Lustre: Skipped 6 previous similar messages [ 1023.172771] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 10:39:23 (1787841563) [ 1040.866159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1040.878084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1040.915708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.893831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1041.897791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1041.915077] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1044.167681] Lustre: server umount lustre-OST0000 complete [ 1051.615973] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1051.629149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1051.645732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1051.655552] Lustre: Skipped 1 previous similar message [ 1057.250740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1057.258971] Lustre: Skipped 3 previous similar messages [ 1062.367706] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1062.372384] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.396751] Lustre: Skipped 1 previous similar message [ 1062.707939] Lustre: server umount lustre-MDT0000 complete [ 1066.465879] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1066.488654] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1066.508662] LustreError: 35745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841609 with bad export cookie 3019222957937636651 [ 1066.514303] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1066.528710] LustreError: 35745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1067.093874] Lustre: server umount lustre-MDT0001 complete [ 1078.792168] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 1081.499294] Key type lgssc unregistered [ 1081.805755] LNet: 42870:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1081.814890] LNetError: 42870:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1081.833126] LNet: Removed LNI 192.168.206.124@tcp [ 1082.520777] Key type .llcrypt unregistered [ 1082.522583] Key type ._llcrypt unregistered [ 1100.954672] Key type ._llcrypt registered [ 1100.956863] Key type .llcrypt registered [ 1101.040875] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 1114.952871] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 1116.421277] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1116.515989] alg: No test for adler32 (adler32-zlib) [ 1117.667402] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1117.953220] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 1119.599478] Key type lgssc registered [ 1120.569645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1155.514552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1155.561178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1156.754150] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1156.780535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1156.848436] Lustre: lustre-MDT0000: new disk, initializing [ 1156.924183] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1156.943649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1161.477501] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1171.319852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1171.470430] Lustre: 46021:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1171.561276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1171.572313] Lustre: Skipped 1 previous similar message [ 1171.656166] Lustre: lustre-MDT0001: new disk, initializing [ 1171.765125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1171.856909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1171.871370] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1176.067650] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1186.228582] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1191.915342] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1198.158151] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1198.491246] Lustre: lustre-OST0000: new disk, initializing [ 1198.495861] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1198.501963] Lustre: 47987:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1198.589206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1204.458837] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1204.801567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1204.821931] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1204.950581] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1214.884968] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1219.958583] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1220.262103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1225.426602] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1225.629225] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1230.820307] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1230.835882] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1230.849133] LustreError: Skipped 1 previous similar message [ 1230.868502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1230.889778] Lustre: Skipped 1 previous similar message [ 1233.904880] Lustre: server umount lustre-OST0000 complete [ 1238.500905] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1238.507977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1238.530828] Lustre: Skipped 1 previous similar message [ 1238.544654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.060485] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1241.066045] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1244.673110] Lustre: server umount lustre-MDT0000 complete [ 1248.328648] LustreError: 46014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841791 with bad export cookie 12576086706100045692 [ 1248.346432] LustreError: 46014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1248.348406] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1248.840656] Lustre: server umount lustre-MDT0001 complete [ 1259.239679] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 10:43:19 (1787841799) [ 1266.450457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1267.112096] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1267.143746] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1267.276850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1271.849481] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1272.288848] LustreError: 50448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1276.390802] LustreError: 50447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1281.247360] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1281.506154] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1281.516586] Lustre: Skipped 1 previous similar message [ 1281.690258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1286.307615] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1296.275377] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1302.140542] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1309.638171] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.281319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1317.087551] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1325.430389] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1331.393531] LustreError: 52977:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1332.585153] Lustre: Modifying parameter general.timeout=1 in log params [ 1333.670146] Lustre: Modifying parameter general.timeout=2 in log params [ 1336.219867] Lustre: Modifying parameter general.timeout=4 in log params [ 1336.225994] Lustre: Skipped 1 previous similar message [ 1341.015288] Lustre: Modifying parameter general.timeout=8 in log params [ 1341.029238] Lustre: Skipped 3 previous similar messages [ 1349.249328] Lustre: Modifying parameter general.timeout=17 in log params [ 1349.253716] Lustre: Skipped 8 previous similar messages [ 1365.904419] Lustre: Modifying parameter general.timeout=35 in log params [ 1365.909197] Lustre: Skipped 17 previous similar messages [ 1398.551866] Lustre: Modifying parameter general.timeout=76 in log params [ 1398.557314] Lustre: Skipped 40 previous similar messages [ 1462.489290] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 10:46:42 (1787842002) [ 1463.671492] Lustre: Setting parameter general.ost.OSS.ost_io.nrs_policies=tbf in log params [ 1463.678019] Lustre: Skipped 54 previous similar messages [ 1465.728721] Lustre: 59923:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1467.526624] Lustre: 60031:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1471.475593] Lustre: 60230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1471.486403] Lustre: 60230:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1502.429066] Lustre: 61802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1502.435206] Lustre: 61802:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1502.449950] LustreError: 61802:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1506.726933] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 10:47:27 (1787842047) [ 1522.144808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1522.160539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1522.186674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1526.562812] Lustre: server umount lustre-MDT0000 complete [ 1529.827115] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1529.855125] LustreError: 50443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1530.186557] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842072 with bad export cookie 12576086706100046574 [ 1530.187790] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1530.195502] LustreError: 50428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1530.661019] Lustre: server umount lustre-MDT0001 complete [ 1545.201819] Lustre: server umount lustre-OST0000 complete [ 1570.961057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1571.324441] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1571.582622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1576.069349] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1584.084548] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1584.206988] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1584.233540] Lustre: 64333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1588.823727] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1595.614094] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1600.849275] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1606.658647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1606.802331] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1607.130522] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.149440] Lustre: Skipped 1 previous similar message [ 1613.239760] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1619.669897] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1621.158622] Lustre: Setting parameter lustre.mdt.lustre-MDT0000.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log params [ 1621.165515] Lustre: Skipped 22 previous similar messages [ 1621.210339] Lustre: 66902:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1629.667593] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1629.671716] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1629.684429] LustreError: Skipped 1 previous similar message [ 1629.694328] Lustre: Skipped 2 previous similar messages [ 1629.695896] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1629.709953] Lustre: Skipped 4 previous similar messages [ 1633.040442] Lustre: server umount lustre-OST0000 complete [ 1639.905045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.920784] Lustre: Skipped 2 previous similar messages [ 1642.894863] Lustre: server umount lustre-MDT0000 complete [ 1645.025279] LustreError: 64345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.044637] LustreError: 64345:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1646.384539] LustreError: 65085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842189 with bad export cookie 12576086706100049703 [ 1646.391310] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1646.400838] LustreError: 65085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1646.734641] Lustre: server umount lustre-MDT0001 complete [ 1657.157342] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 1659.595380] Key type lgssc unregistered [ 1659.857629] LNet: 68114:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1659.867599] LNetError: 68114:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1659.895373] LNet: Removed LNI 192.168.206.124@tcp [ 1660.638094] Key type .llcrypt unregistered [ 1660.641584] Key type ._llcrypt unregistered [ 1684.315411] Key type ._llcrypt registered [ 1684.317505] Key type .llcrypt registered [ 1684.433789] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 1685.388497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1685.470339] alg: No test for adler32 (adler32-zlib) [ 1686.570252] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1686.747579] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 1688.391244] Key type lgssc registered [ 1689.268532] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1696.376466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1696.404565] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.810921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.128679] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1708.007420] LustreError: 69527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1708.021644] LustreError: 69527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1708.630223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1709.173472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1713.266624] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1719.831896] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1724.723660] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1730.957698] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1731.571540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.202828] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1744.035392] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1752.426512] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1760.224510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1760.230867] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1760.246496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1763.299954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1763.305246] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1763.317088] Lustre: Skipped 1 previous similar message [ 1765.866629] Lustre: server umount lustre-MDT0000 complete [ 1768.417617] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1768.432324] LustreError: 69528:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1769.086644] LustreError: 69508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842311 with bad export cookie 11675005979852256598 [ 1769.088632] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1769.100727] LustreError: 69508:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1769.337682] Lustre: server umount lustre-MDT0001 complete [ 1782.514792] Lustre: server umount lustre-OST0000 complete [ 1809.640197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.001713] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1810.036646] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1810.312455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1814.466343] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1822.561149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.700809] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1822.734947] Lustre: 74399:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1823.306818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1827.433852] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1835.121965] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1840.758500] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1847.937707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1848.098954] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1848.430297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1854.492225] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1862.295569] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1863.906783] Lustre: 76972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1865.781409] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1870.305780] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1870.313577] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1870.322513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1870.326579] Lustre: Skipped 2 previous similar messages [ 1874.913514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1874.915800] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1874.925173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1880.034457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1880.045893] Lustre: Skipped 2 previous similar messages [ 1884.127467] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1884.286541] Lustre: server umount lustre-OST0000 complete [ 1889.767307] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1889.780282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1889.815145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1894.727880] Lustre: server umount lustre-MDT0000 complete [ 1895.392965] LustreError: 74409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1895.416796] LustreError: 74409:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1899.367041] LustreError: 74391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842442 with bad export cookie 11675005979852257445 [ 1899.368129] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1899.400656] LustreError: 74391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1899.848080] Lustre: server umount lustre-MDT0001 complete [ 1911.239705] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 1913.705477] Key type lgssc unregistered [ 1913.906164] LNet: 78186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1913.910967] LNetError: 78186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1913.923833] LNet: Removed LNI 192.168.206.124@tcp [ 1914.718760] Key type .llcrypt unregistered [ 1914.728839] Key type ._llcrypt unregistered [ 1940.156547] Key type ._llcrypt registered [ 1940.163679] Key type .llcrypt registered [ 1940.339794] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 1942.201629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.265614] alg: No test for adler32 (adler32-zlib) [ 1943.665574] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 1944.070846] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 1945.832481] Key type lgssc registered [ 1947.669844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1957.866334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1957.902500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1959.268472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1964.119963] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1968.622913] LustreError: 79601:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1968.664574] LustreError: 79601:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1973.728144] LustreError: 79602:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1973.880669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1974.667034] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1979.454279] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 1986.490060] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1992.838687] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1999.407753] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2000.096391] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2006.625575] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2014.713499] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2025.872870] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2026.932906] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2027.982499] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2030.765946] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2030.772406] Lustre: Skipped 1 previous similar message [ 2035.198588] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2035.204272] Lustre: Skipped 3 previous similar messages [ 2043.776765] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2043.786453] Lustre: Skipped 8 previous similar messages [ 2060.582473] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2060.586829] Lustre: Skipped 17 previous similar messages [ 2093.500134] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2093.506471] Lustre: Skipped 34 previous similar messages [ 2128.974891] Lustre: 87712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2131.895774] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 10:57:52 (1787842672) [ 2133.376894] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2135.378629] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 10:57:55 (1787842675) [ 2142.689938] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.693401] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2142.729022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2142.750513] Lustre: Skipped 1 previous similar message [ 2145.813036] Lustre: server umount lustre-OST0000 complete [ 2152.929532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2152.937527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2152.953235] Lustre: Skipped 2 previous similar messages [ 2152.968445] Lustre: Skipped 1 previous similar message [ 2158.049800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.169848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2163.179051] Lustre: Skipped 1 previous similar message [ 2164.192588] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2164.423376] Lustre: server umount lustre-MDT0000 complete [ 2168.289325] LustreError: 80332:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.318678] LustreError: 80332:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2168.443120] LustreError: 82885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842711 with bad export cookie 17075140555560146110 [ 2168.452438] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2168.457729] LustreError: 82885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2168.699814] Lustre: server umount lustre-MDT0001 complete [ 2181.010220] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 2183.811784] Key type lgssc unregistered [ 2184.098771] LNet: 88964:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2184.107175] LNetError: 88964:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2185.127622] LNet: Removed LNI 192.168.206.124@tcp [ 2185.982524] Key type .llcrypt unregistered [ 2185.986513] Key type ._llcrypt unregistered [ 2190.142206] Key type ._llcrypt registered [ 2190.144394] Key type .llcrypt registered [ 2190.242692] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2194.818558] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules [ 2195.705333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2195.753058] alg: No test for adler32 (adler32-zlib) [ 2196.899404] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 2197.181917] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 2198.863315] Key type lgssc registered [ 2200.081469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2201.391722] LustreError: 90166:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2220.433635] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 2223.946865] LustreError: 90166:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2230.875200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2230.905188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2232.418559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2236.827371] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2243.050162] LustreError: 91176:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2243.084392] LustreError: 91176:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2245.608803] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 10:59:45 (1787842785) [ 2248.172632] LustreError: 91177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2250.832212] LustreError: 91208:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2250.849767] LustreError: 91208:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2250.858401] LustreError: 91208:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 0, failed: rc = -5 [ 2251.135514] Lustre: server umount lustre-MDT0000 complete [ 2264.782670] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 2267.372775] Key type lgssc unregistered [ 2267.638760] LNet: 92704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2267.643437] LNetError: 92704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2268.709692] LNet: Removed LNI 192.168.206.124@tcp [ 2269.576332] Key type .llcrypt unregistered [ 2269.579113] Key type ._llcrypt unregistered [ 2287.176351] Key type ._llcrypt registered [ 2287.179071] Key type .llcrypt registered [ 2287.387659] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 2289.069977] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2289.234170] alg: No test for adler32 (adler32-zlib) [ 2290.220265] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 2290.414447] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 2292.079162] Key type lgssc registered [ 2293.441114] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2303.186292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2303.203261] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2304.865410] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2309.698883] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2315.239126] LustreError: 93923:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2315.256079] LustreError: 93923:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2318.875952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2319.806059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2324.329817] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2332.587162] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2338.982715] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2345.878276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2346.434407] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2352.443403] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2360.032930] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2411.352877] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 11:02:31 (1787842951) [ 2413.495577] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2415.228936] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 11:02:35 (1787842955) [ 2421.224021] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.229900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2421.247695] Lustre: Skipped 1 previous similar message [ 2421.271631] Lustre: Skipped 2 previous similar messages [ 2426.345682] LustreError: 95183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.388850] LustreError: 95183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2426.520580] Lustre: server umount lustre-MDT0000 complete [ 2430.222299] LustreError: 93902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842972 with bad export cookie 7487236432323068446 [ 2430.227120] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2430.239120] LustreError: 93902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2430.573805] Lustre: server umount lustre-MDT0001 complete [ 2445.475676] Lustre: server umount lustre-OST0000 complete [ 2458.200306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2458.710077] LustreError: 98232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2458.951359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2464.230382] LustreError: 98233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2464.849593] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2468.324653] LustreError: 98232:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2474.501490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2475.191351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2479.556497] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2486.890558] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2492.918618] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2505.940259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2506.173950] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2506.178318] LustreError: 98222:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2506.198588] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2506.212574] LustreError: 100316:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2506.224764] LustreError: 100316:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2506.231250] LustreError: 100316:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2506.312453] Lustre: server umount lustre-OST0000 complete [ 2506.314965] LustreError: 100316:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2514.354731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2514.521685] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2514.528924] LustreError: 98222:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2514.555202] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2514.562851] LustreError: 100781:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -98 [ 2514.568344] LustreError: 100781:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 2514.574262] LustreError: 100781:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2514.659536] Lustre: server umount lustre-OST0000 complete [ 2514.676434] LustreError: 100781:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2524.450761] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2524.719717] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2524.726790] Lustre: Found index 0 for lustre-OST0000, updating log [ 2524.735214] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2524.853191] Lustre: lustre-OST0000: new disk, initializing [ 2524.864527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2524.871731] Lustre: 101318:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2525.064837] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2531.402324] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2534.875893] Lustre: Failing over lustre-OST0000 [ 2535.045453] Lustre: server umount lustre-OST0000 complete [ 2539.489678] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2539.502111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2539.507116] Lustre: Skipped 2 previous similar messages [ 2539.525913] Lustre: Skipped 1 previous similar message [ 2544.610082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2544.630490] Lustre: Skipped 1 previous similar message [ 2548.930156] Lustre: server umount lustre-MDT0000 complete [ 2549.736473] LustreError: 98229:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2549.751674] LustreError: 98229:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2553.334412] LustreError: 98213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843096 with bad export cookie 7487236432323071190 [ 2553.343710] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.355563] LustreError: 98213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2553.718506] Lustre: server umount lustre-MDT0001 complete [ 2563.407200] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 11:05:03 (1787843103) [ 2579.094906] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 2589.675220] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2590.049470] LustreError: 103663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2590.175821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.230294] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2602.134872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2606.641778] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2609.376807] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2615.859426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2616.524482] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2621.385761] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2627.815391] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2627.828492] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 [ 2627.861959] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2629.365983] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2629.484468] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2629.606912] Lustre: lustre-OST0001: new disk, initializing [ 2629.611667] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2629.617139] Lustre: 105981:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2629.829497] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:57 [ 2629.848288] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2629.869281] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2629.876114] Lustre: Skipped 2 previous similar messages [ 2634.992420] Lustre: lustre-OST0000: Denying connection for new client lustre-MDT0001-mdtlov_UUID (at 0@lo), waiting for 1 known clients (0 recovered, 1 in progress, and 0 evicted) to recover in 1:02 [ 2635.017520] LustreError: lustre-OST0000-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -16 [ 2635.046646] Lustre: lustre-OST0000: Recovery over after 0:08, of 1 clients 1 recovered and 0 were evicted. [ 2635.046935] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 2635.060278] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:195 to 0x280000401:226) [ 2636.270757] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2638.379464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2638.391256] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2638.488685] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2643.265334] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2648.758651] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2654.184358] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2654.191348] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2654.201657] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2654.204365] Lustre: Skipped 2 previous similar messages [ 2654.239747] Lustre: Skipped 2 previous similar messages [ 2658.823888] Lustre: server umount lustre-MDT0001 complete [ 2659.305531] LustreError: 103663:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.326304] LustreError: 103663:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2667.892098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.060406] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2668.066282] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2668.127789] Lustre: 103652:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2669.557600] LustreError: 107525:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2669.564803] LustreError: 107525:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2669.569770] LustreError: 103656:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2669.654289] LustreError: 107535:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.124@tcp: cfg command failed: rc = -17 [ 2669.661114] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2669.661114] [ 2673.402547] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2674.166591] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2683.290942] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 11:07:03 (1787843223) [ 2700.651610] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 2717.302801] Lustre: 109597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2736.644742] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2740.054700] Lustre: 110733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2827.237810] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2827.250048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.262456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.265651] Lustre: Skipped 1 previous similar message [ 2830.424861] Lustre: server umount lustre-MDT0000 complete [ 2832.865189] LustreError: 104734:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2832.894618] LustreError: 104734:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2834.328453] LustreError: 103645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843377 with bad export cookie 7487236432323072191 [ 2834.329517] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2834.345969] LustreError: 103645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2834.610897] Lustre: server umount lustre-MDT0001 complete [ 2848.901967] Lustre: server umount lustre-OST0000 complete [ 2863.275912] Lustre: server umount lustre-OST0001 complete [ 2868.148450] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2876.781846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2884.080854] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2892.464725] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2909.996705] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 2918.152452] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2918.170981] Lustre: lustre-MDT0000: reset Object Index mappings [ 2918.646461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2918.649961] Lustre: Skipped 1 previous similar message [ 2922.060864] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2929.954079] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2929.986867] Lustre: lustre-MDT0001: reset Object Index mappings [ 2934.858825] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2937.841582] Lustre: 115844:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2944.884764] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2952.418883] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2956.072407] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 2961.401836] LustreError: 116198:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2961.426437] LustreError: 116198:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2961.476309] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2964.141635] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:387 to 0x280000401:418) [ 2964.152099] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2964.155307] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2968.053445] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 2976.089368] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2979.805205] Lustre: 117723:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 3026.984697] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 11:12:47 (1787843567) [ 3053.537380] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3053.549615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3053.562922] Lustre: Skipped 3 previous similar messages [ 3053.567815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3053.572847] Lustre: Skipped 3 previous similar messages [ 3059.729366] Lustre: server umount lustre-MDT0000 complete [ 3062.668581] LustreError: 115444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843605 with bad export cookie 7487236432323119266 [ 3062.678153] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3062.680759] LustreError: 115444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3066.849192] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3066.857427] Lustre: Skipped 4 previous similar messages [ 3069.013147] Lustre: server umount lustre-MDT0001 complete [ 3086.304077] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3086.431224] Lustre: server umount lustre-OST0000 complete [ 3095.699992] Lustre: server umount lustre-OST0001 complete [ 3101.519465] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 3110.320708] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 3145.127636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3145.324717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3145.361195] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3145.462344] Lustre: lustre-MDT0000: new disk, initializing [ 3145.609736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3145.631840] Lustre: Skipped 3 previous similar messages [ 3145.648690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3150.741622] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 3158.994745] Lustre: server umount lustre-MDT0000 complete [ 3168.869984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3169.194265] Lustre: Found index 0 for lustre-MDT0000, updating log [ 3169.207071] Lustre: 121988:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3169.226108] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 3174.523450] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 3179.849970] Lustre: server umount lustre-MDT0000 complete [ 3188.961672] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 11:15:29 (1787843729) [ 3190.817837] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 3192.845482] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 11:15:33 (1787843733) [ 3210.268346] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 3220.149148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3226.855678] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 3237.022593] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3237.097895] Lustre: 124038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3237.145471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3237.148687] Lustre: Skipped 1 previous similar message [ 3237.199967] Lustre: lustre-MDT0001: new disk, initializing [ 3237.371232] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3237.387495] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3242.421356] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 3247.332419] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 3254.409400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3254.732947] Lustre: lustre-OST0000: new disk, initializing [ 3254.738431] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3254.755616] Lustre: 125618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3256.623433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3256.637091] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3256.748605] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3260.821925] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 3270.919667] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3271.080730] Lustre: lustre-OST0001: new disk, initializing [ 3271.083585] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3271.088388] Lustre: 126489:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3273.008375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3273.047455] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3273.162485] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3277.541286] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 3288.353580] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3292.579506] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3666.960321] Lustre: 132906:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3666.968514] Lustre: 132906:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3666.974471] Lustre: 132906:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3666.987662] Lustre: 132906:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3666.998636] Lustre: 132906:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3667.009591] Lustre: 132906:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3941.639935] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 11:28:01 (1787844481) [ 3959.265674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.283373] Lustre: Skipped 2 previous similar messages [ 3959.284515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3959.308556] Lustre: Skipped 11 previous similar messages [ 3962.436222] Lustre: server umount lustre-MDT0000 complete [ 3964.389372] LustreError: 124046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.403139] LustreError: 124046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 3965.768219] LustreError: 124032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844508 with bad export cookie 7487236432323187047 [ 3965.768993] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.786160] LustreError: 124032:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3969.505226] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.511841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3969.528890] Lustre: Skipped 1 previous similar message [ 3969.544350] Lustre: Skipped 1 previous similar message [ 3975.648881] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3975.654667] Lustre: Skipped 3 previous similar messages [ 3980.255152] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3980.564497] Lustre: server umount lustre-MDT0001 complete [ 3990.574556] Lustre: server umount lustre-OST0000 complete [ 4000.117624] Lustre: server umount lustre-OST0001 complete [ 4006.466357] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 4016.496200] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 4052.557075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4052.848349] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4052.890463] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4052.968245] Lustre: lustre-MDT0000: new disk, initializing [ 4053.030300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4053.041221] Lustre: Skipped 5 previous similar messages [ 4053.054592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4057.614753] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 4069.717884] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4069.916050] Lustre: 135958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4069.992315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4069.996771] Lustre: Skipped 1 previous similar message [ 4070.086974] Lustre: lustre-MDT0001: new disk, initializing [ 4070.188758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4070.199913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4074.769317] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 4084.173853] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4089.855299] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4096.925406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4097.264787] Lustre: lustre-OST0000: new disk, initializing [ 4097.268220] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4097.276124] Lustre: 137927:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4097.355665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4097.369337] Lustre: Skipped 1 previous similar message [ 4098.887624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4098.906590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4099.035564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4103.368871] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 4112.981076] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4122.375185] Lustre: lustre-MDD0000: changelog on [ 4124.801874] Lustre: lustre-MDD0001: changelog on [ 6079.534934] Lustre: lustre-MDD0001: changelog off [ 6081.563744] Lustre: 141631:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 6082.482489] Lustre: lustre-MDD0000: changelog off [ 6090.724461] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6090.726824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6090.731196] LustreError: Skipped 1 previous similar message [ 6090.749634] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6090.761754] Lustre: Skipped 3 previous similar messages [ 6093.807756] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6093.811087] Lustre: Skipped 1 previous similar message [ 6096.039889] Lustre: server umount lustre-OST0000 complete [ 6103.009425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6103.024755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6103.041030] Lustre: Skipped 1 previous similar message [ 6103.046120] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6104.034172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6107.203637] Lustre: server umount lustre-MDT0000 complete [ 6109.157524] LustreError: 139562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.181783] LustreError: 139562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6111.822149] LustreError: 135948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846654 with bad export cookie 7487236432324008994 [ 6111.836798] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6111.843571] LustreError: 135948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6112.616660] Lustre: server umount lustre-MDT0001 complete [ 6127.663854] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 6131.653313] Key type lgssc unregistered [ 6132.017865] LNet: 147083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6132.023549] LNetError: 147083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6132.038643] LNet: Removed LNI 192.168.206.124@tcp [ 6133.339351] Key type .llcrypt unregistered [ 6133.344635] Key type ._llcrypt unregistered [ 6154.314249] Key type ._llcrypt registered [ 6154.317701] Key type .llcrypt registered [ 6154.437520] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 6171.284346] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 6172.483728] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6172.529582] alg: No test for adler32 (adler32-zlib) [ 6173.560191] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 6173.774782] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 6175.431204] Key type lgssc registered [ 6176.716600] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6214.833977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6214.873123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6216.235941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6216.295379] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6216.410486] Lustre: lustre-MDT0000: new disk, initializing [ 6216.564992] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6216.589898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6221.745830] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6234.745350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6234.960481] Lustre: 150287:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6235.166057] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6235.178323] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6235.200284] Lustre: Skipped 1 previous similar message [ 6235.363098] Lustre: lustre-MDT0001: new disk, initializing [ 6235.512472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6235.575766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6235.602740] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6241.832759] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6253.802964] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6261.530791] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6269.453890] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6269.791224] Lustre: lustre-OST0000: new disk, initializing [ 6269.798551] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6269.807240] Lustre: 152256:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6269.987292] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6276.719615] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6276.744562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6276.914798] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6277.421881] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6287.212429] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6295.873906] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_module obdecho/obdecho [ 6298.731247] LustreError: 153260:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6298.744156] LustreError: 153260:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6300.936328] LustreError: 153358:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6317.538567] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6317.556333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6317.588467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6320.314609] Lustre: server umount lustre-MDT0000 complete [ 6324.515137] LustreError: 150278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787846867 with bad export cookie 4531241955808975743 [ 6324.522726] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6324.525103] LustreError: 150278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6324.963322] Lustre: server umount lustre-MDT0001 complete [ 6339.459305] Lustre: server umount lustre-OST0000 complete [ 6361.212945] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 6372.849330] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 6412.422949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.640430] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6412.662468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6412.749469] Lustre: lustre-MDT0000: new disk, initializing [ 6412.847238] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6412.870279] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6417.516752] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6428.275536] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6428.375183] Lustre: 157075:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6428.446289] LustreError: 157086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6428.480228] LustreError: 157086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6428.483281] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6428.508285] Lustre: Skipped 1 previous similar message [ 6428.613599] Lustre: lustre-MDT0001: new disk, initializing [ 6428.736216] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6428.791983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6428.804775] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6433.389283] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6443.359726] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6449.265878] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6455.591785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6455.888988] Lustre: lustre-OST0000: new disk, initializing [ 6455.893974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6455.906364] Lustre: 159041:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6456.004595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6461.019957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6461.037983] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6461.181767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6462.593516] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6475.053476] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6482.226026] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6482.464911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6487.471360] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6487.672336] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6492.138617] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6492.140534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6492.149751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6492.149759] Lustre: Skipped 3 previous similar messages [ 6492.151132] LustreError: Skipped 1 previous similar message [ 6492.166557] Lustre: Skipped 3 previous similar messages [ 6496.069092] Lustre: server umount lustre-OST0000 complete [ 6502.377576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6502.380982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6502.385089] Lustre: Skipped 1 previous similar message [ 6506.723045] Lustre: server umount lustre-MDT0000 complete [ 6507.488931] LustreError: 157087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6507.514389] LustreError: 157087:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6510.281368] LustreError: 157066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847053 with bad export cookie 4531241955808976709 [ 6510.287311] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6510.295533] LustreError: 157066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6510.618879] Lustre: server umount lustre-MDT0001 complete [ 6513.690722] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 12:10:54 (1787847054) [ 6515.477331] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6517.268650] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 12:10:57 (1787847057) [ 6525.029058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6525.557180] LustreError: 161453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6525.655677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6530.885255] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6531.047977] LustreError: 161454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6536.161036] LustreError: 161453:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6540.686062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.705877] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6553.902555] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6560.463593] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6568.236076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6568.489263] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6568.496609] Lustre: Skipped 1 previous similar message [ 6575.070059] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6584.270763] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6600.736132] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.24@tcp (stopping) [ 6600.743498] Lustre: Skipped 1 previous similar message [ 6601.186817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.213129] Lustre: Skipped 1 previous similar message [ 6605.889378] Lustre: server umount lustre-MDT0000 complete [ 6606.305934] LustreError: 161449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6606.332845] LustreError: 161449:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6610.015789] LustreError: 161436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847152 with bad export cookie 4531241955808977591 [ 6610.032533] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6610.034824] LustreError: 161436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6611.003256] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.24@tcp (stopping) [ 6611.023920] Lustre: Skipped 4 previous similar messages [ 6611.426093] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6611.484471] Lustre: Skipped 1 previous similar message [ 6616.706173] Lustre: server umount lustre-MDT0001 complete [ 6625.413651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6639.073057] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6639.357265] LustreError: 165317:0:(llog.c:1655:llog_backup()) MGC192.168.206.124@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6639.370197] Lustre: 165317:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.124@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6640.098019] LustreError: 165322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6640.124510] LustreError: 165322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6674.912155] LustreError: 165323:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.939618] LustreError: 165323:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [ 6700.517365] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6700.536356] Lustre: 165317:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.124@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6740.450513] LustreError: 165322:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6740.469941] LustreError: 165322:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 6741.494909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6746.609957] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6748.341627] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6756.705951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6770.145554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -19 [ 6779.360844] LustreError: 148509:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000646652f9 x1874693158217344/t0(0) o250->MGC192.168.206.124@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6784.433798] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6785.010092] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6791.658475] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6793.377135] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6798.675937] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6800.279473] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6820.837516] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6820.856357] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6820.857243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6820.857249] Lustre: Skipped 3 previous similar messages [ 6820.863027] Lustre: Skipped 1 previous similar message [ 6825.819892] Lustre: server umount lustre-MDT0000 complete [ 6829.031186] LustreError: 161434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847371 with bad export cookie 4531241955808984080 [ 6829.033785] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6829.048195] LustreError: 161434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6829.360644] Lustre: server umount lustre-MDT0001 complete [ 6843.701474] Lustre: server umount lustre-OST0000 complete [ 6863.938837] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 6873.450964] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 6912.540306] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6912.805099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6912.841535] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6912.941610] Lustre: lustre-MDT0000: new disk, initializing [ 6913.044064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6913.060934] Lustre: Skipped 1 previous similar message [ 6913.097142] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6917.972319] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6930.610829] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6930.750498] Lustre: 170435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6931.046490] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6935.991146] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6945.965287] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6952.281577] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6959.784788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6960.209622] Lustre: lustre-OST0000: new disk, initializing [ 6960.221094] Lustre: Skipped 1 previous similar message [ 6960.226986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6960.236740] Lustre: Skipped 2 previous similar messages [ 6960.242371] Lustre: 172407:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6966.680523] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 6970.025791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6970.051663] Lustre: Skipped 1 previous similar message [ 6970.075498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6970.220933] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6976.685187] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6981.489819] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6981.755487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6987.895995] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6988.174886] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6990.820828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6990.826369] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6990.827303] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6990.827308] Lustre: Skipped 2 previous similar messages [ 6990.839723] Lustre: Skipped 2 previous similar messages [ 6996.502124] Lustre: server umount lustre-OST0000 complete [ 7006.939489] Lustre: server umount lustre-MDT0000 complete [ 7010.970054] LustreError: 172406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847553 with bad export cookie 4531241955809000873 [ 7010.974927] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7010.984828] LustreError: 172406:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7011.360390] Lustre: server umount lustre-MDT0001 complete [ 7020.063887] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 12:19:20 (1787847560) [ 7026.708370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7027.208216] LustreError: 174817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7027.229494] LustreError: 174817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7031.735453] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7039.924538] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7044.468176] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7051.600588] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7058.048708] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7065.165657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7072.271995] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7080.031396] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7091.687606] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 12:20:32 (1787847632) [ 7097.313257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7097.315162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7097.322535] LustreError: Skipped 1 previous similar message [ 7097.331305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7097.336510] Lustre: Skipped 3 previous similar messages [ 7097.367927] Lustre: Skipped 8 previous similar messages [ 7102.895229] Lustre: server umount lustre-OST0000 complete [ 7112.992849] Lustre: server umount lustre-MDT0000 complete [ 7116.281364] LustreError: 174797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847659 with bad export cookie 4531241955809001755 [ 7116.287450] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7116.291037] LustreError: 174797:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7116.867894] Lustre: server umount lustre-MDT0001 complete [ 7129.180389] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 7131.857475] Key type lgssc unregistered [ 7132.083784] LNet: 178549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7132.093228] LNetError: 178549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7132.105450] LNet: Removed LNI 192.168.206.124@tcp [ 7132.885375] Key type .llcrypt unregistered [ 7132.888165] Key type ._llcrypt unregistered [ 7151.942194] Key type ._llcrypt registered [ 7151.947674] Key type .llcrypt registered [ 7152.061054] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 7153.045117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7153.109269] alg: No test for adler32 (adler32-zlib) [ 7154.192507] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 7154.420474] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 7156.103140] Key type lgssc registered [ 7157.201571] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7164.841948] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7164.872711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7180.511777] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7185.631272] LustreError: 179793:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.124@tcp: failed processing log, type 4: rc = -110 [ 7190.753176] LustreError: 179793:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7190.774099] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7190.783786] LustreError: MGC192.168.206.124@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7190.797036] LustreError: 179793:0:(tgt_mount.c:1885:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7190.813691] LustreError: 179793:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -22 [ 7190.823872] LustreError: 179793:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 7190.904517] Lustre: server umount lustre-OST0000 complete [ 7190.906614] LustreError: 179793:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7199.458081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7200.965154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7205.164137] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7210.466432] LustreError: 180292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7210.487940] LustreError: 180292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7213.221746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.544096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7218.106531] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7226.490433] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7232.717942] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7240.276342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7240.484384] LustreError: 182110:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7240.499915] Lustre: 182110:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.206.124@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7240.656758] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7247.682930] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7255.095730] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7259.234268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7259.242099] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7259.259878] Lustre: Skipped 1 previous similar message [ 7259.277030] Lustre: Skipped 2 previous similar messages [ 7263.567672] Lustre: server umount lustre-MDT0000 complete [ 7264.739090] LustreError: 180291:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7264.765388] LustreError: 180291:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7267.132582] LustreError: 180274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847809 with bad export cookie 6912373163250492791 [ 7267.134274] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7267.155346] LustreError: 180274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7267.448207] LustreError: 181021:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000005c92e00a x1874694186758400/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7267.610253] Lustre: server umount lustre-MDT0001 complete [ 7281.918637] Lustre: server umount lustre-OST0000 complete [ 7292.227967] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 7301.799555] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 7338.773809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7339.108772] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7339.157974] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7339.272273] Lustre: lustre-MDT0000: new disk, initializing [ 7339.394427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.419552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7344.090730] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7354.914161] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.022731] Lustre: 185632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7355.108675] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7355.116673] Lustre: Skipped 1 previous similar message [ 7355.226848] Lustre: lustre-MDT0001: new disk, initializing [ 7355.334698] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7355.383691] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7355.403589] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7359.535884] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7368.919048] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7374.204735] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7380.391656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7380.637603] Lustre: lustre-OST0000: new disk, initializing [ 7380.641495] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7380.647115] Lustre: 187600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7380.726808] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7387.177392] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7389.307929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7389.326295] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7389.452356] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7397.202001] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7402.904514] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7403.120530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7407.566985] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7407.762614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7410.153490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7410.154691] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7410.157554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7410.163502] LustreError: Skipped 1 previous similar message [ 7410.197616] Lustre: Skipped 2 previous similar messages [ 7415.272279] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7415.281743] Lustre: Skipped 2 previous similar messages [ 7415.800764] Lustre: server umount lustre-OST0000 complete [ 7420.388426] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7420.395763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7420.410907] Lustre: Skipped 1 previous similar message [ 7425.504520] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7425.522516] Lustre: Skipped 2 previous similar messages [ 7434.207189] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7434.821529] Lustre: server umount lustre-MDT0000 complete [ 7435.746667] LustreError: 185645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7435.789251] LustreError: 185645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7439.454494] LustreError: 185625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787847982 with bad export cookie 6912373163250493512 [ 7439.460105] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7439.472274] LustreError: 185625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7440.135339] Lustre: server umount lustre-MDT0001 complete [ 7451.794543] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 12:26:31 (1787847991) [ 7471.106986] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 7473.932621] Key type lgssc unregistered [ 7474.241732] LNet: 190550:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7474.251122] LNetError: 190550:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7475.303779] LNet: Removed LNI 192.168.206.124@tcp [ 7476.201136] Key type .llcrypt unregistered [ 7476.208684] Key type ._llcrypt unregistered [ 7496.999399] Key type ._llcrypt registered [ 7497.005099] Key type .llcrypt registered [ 7497.139145] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 7498.491737] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7498.547890] alg: No test for adler32 (adler32-zlib) [ 7499.688544] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 7499.982070] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 7501.663117] Key type lgssc registered [ 7502.749296] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7511.484393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7511.497820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7512.779392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7517.292707] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7522.274185] LustreError: 191776:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7522.296037] LustreError: 191776:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7527.397208] LustreError: 191777:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7528.535063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7528.929828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7534.568335] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7543.330509] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7550.478486] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7559.897751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7560.129787] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7560.132950] LustreError: 193642:0:(llog.c:1696:llog_backup()) MGC192.168.206.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7560.148645] Lustre: 193642:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.206.124@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7560.266186] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.941803] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7576.482825] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7578.594452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.599352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7578.614715] Lustre: Skipped 2 previous similar messages [ 7578.623435] Lustre: Skipped 2 previous similar messages [ 7583.716488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7583.730320] Lustre: Skipped 2 previous similar messages [ 7585.066907] Lustre: server umount lustre-MDT0000 complete [ 7588.836023] LustreError: 192505:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7588.843921] LustreError: 191757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848131 with bad export cookie 11811198751998705001 [ 7588.851353] LustreError: 192505:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7588.855490] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7588.858861] LustreError: 191757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7589.178889] Lustre: server umount lustre-MDT0001 complete [ 7603.148185] Lustre: server umount lustre-OST0000 complete [ 7614.355947] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 7629.270094] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 7673.301204] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7673.555257] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7673.596945] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7673.693452] Lustre: lustre-MDT0000: new disk, initializing [ 7673.775579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7673.796794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7678.218495] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7689.209421] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7689.406411] Lustre: 197181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7689.508320] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7689.514440] Lustre: Skipped 1 previous similar message [ 7689.520343] LustreError: 197963:0:(llog.c:1696:llog_backup()) MGC192.168.206.124@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7689.534204] LustreError: 197963:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7689.549074] Lustre: 197963:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.124@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7689.587962] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7689.597620] Lustre: Skipped 1 previous similar message [ 7689.720803] Lustre: lustre-MDT0001: new disk, initializing [ 7689.834427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7689.864971] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7689.876044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7694.773718] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7704.942626] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7710.799330] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7717.483569] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.714938] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7717.717747] Lustre: Skipped 1 previous similar message [ 7717.720571] LustreError: 199124:0:(llog.c:1696:llog_backup()) MGC192.168.206.124@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7717.727322] LustreError: 199124:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7717.730871] Lustre: 199124:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.206.124@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7717.744391] Lustre: 199124:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7717.778804] Lustre: lustre-OST0000: new disk, initializing [ 7717.786141] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7717.798566] Lustre: 199148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7717.880629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7723.567341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7723.589563] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7723.714234] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7723.996833] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7733.885681] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7739.439748] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7739.749345] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7744.612869] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7744.897816] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7749.600347] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7749.604361] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7749.613526] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7749.615135] Lustre: Skipped 1 previous similar message [ 7749.628019] Lustre: Skipped 1 previous similar message [ 7754.729668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7754.740049] Lustre: Skipped 1 previous similar message [ 7759.844956] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7759.851679] Lustre: Skipped 1 previous similar message [ 7761.375558] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7761.552584] Lustre: server umount lustre-OST0000 complete [ 7766.509965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7766.520693] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.083207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7770.087747] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7770.101416] Lustre: Skipped 1 previous similar message [ 7771.943856] Lustre: server umount lustre-MDT0000 complete [ 7775.209464] LustreError: 197188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7775.242090] LustreError: 197188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7776.130232] LustreError: 199147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848318 with bad export cookie 11811198751998705715 [ 7776.137039] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7776.141597] LustreError: 199147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7776.524857] Lustre: server umount lustre-MDT0001 complete [ 7786.154821] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 12:32:06 (1787848326) [ 7804.092405] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 7815.575520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7815.995315] LustreError: 202278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7816.045148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.094723] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7825.399465] LustreError: 202279:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7825.447512] LustreError: 202279:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7829.947591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7834.568652] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7837.809389] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7845.319748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7852.693366] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7861.709738] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7862.035919] Lustre: lustre-OST0001: new disk, initializing [ 7862.041026] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7862.058076] Lustre: 204587:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7862.187097] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7862.194063] Lustre: Skipped 2 previous similar messages [ 7867.514588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7867.536590] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7867.669265] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7869.829442] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7880.693788] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7890.798757] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7892.850235] Lustre: DEBUG MARKER: ADD OST3 [ 7897.901239] loop: module loaded [ 7898.010050] loop0: detected capacity change from 0 to 204800000 [ 7898.396031] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7898.416226] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7898.428623] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7904.775363] loop0: detected capacity change from 0 to 204800000 [ 7906.142803] Lustre: DEBUG MARKER: STOP OST3 [ 7911.496619] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7911.715148] Lustre: lustre-OST0003: new disk, initializing [ 7911.719684] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7911.726893] Lustre: 206581:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7913.595277] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7913.689901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7913.710042] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7913.755806] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7915.618245] LustreError: 206771:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7916.099778] Lustre: Failing over lustre-OST0003 [ 7916.334056] Lustre: server umount lustre-OST0003 complete [ 7918.085149] LustreError: 204595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 192.168.206.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7918.914742] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7919.073947] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7919.089962] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7919.123658] Lustre: Skipped 1 previous similar message [ 7927.605326] Lustre: DEBUG MARKER: START OST3 again [ 7934.427044] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7934.746701] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7934.755862] Lustre: Skipped 1 previous similar message [ 7934.774499] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7935.978775] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7936.832530] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7936.847173] Lustre: lustre-OST0003-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7940.779540] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 7962.523019] Lustre: Failing over lustre-OST0003 [ 7962.593971] LustreError: lustre-OST0003-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7962.602198] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7962.616766] LustreError: 204595:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.635489] LustreError: 204595:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 7962.673298] Lustre: server umount lustre-OST0003 complete [ 7981.025940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7981.033182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7981.048690] Lustre: Skipped 4 previous similar messages [ 7981.063515] Lustre: Skipped 4 previous similar messages [ 7994.337282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7994.683138] Lustre: server umount lustre-MDT0000 complete [ 7998.321890] LustreError: 202258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848541 with bad export cookie 11811198751998706597 [ 7998.337175] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7998.345284] LustreError: 202258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8004.906080] Lustre: server umount lustre-MDT0001 complete [ 8008.987295] Lustre: server umount lustre-OST0000 complete [ 8013.312306] Lustre: server umount lustre-OST0001 complete [ 8020.387815] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 8028.601852] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 8066.256625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.504961] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8066.537993] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8066.622322] Lustre: lustre-MDT0000: new disk, initializing [ 8066.700150] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8066.717571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8070.785852] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8083.738875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8083.979752] Lustre: 212048:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8084.047168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8084.051134] Lustre: Skipped 1 previous similar message [ 8084.150785] Lustre: lustre-MDT0001: new disk, initializing [ 8084.339574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8084.352309] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8089.676332] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8099.442306] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8104.317428] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8111.748273] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8111.993525] Lustre: 214016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8113.472123] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8113.629814] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8119.177645] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8128.906196] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8134.448574] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8134.693216] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8139.639313] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8139.893875] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8144.362050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8144.365226] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8144.373255] LustreError: Skipped 2 previous similar messages [ 8144.375724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8144.397130] Lustre: Skipped 3 previous similar messages [ 8144.419944] Lustre: Skipped 13 previous similar messages [ 8156.127186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8156.334831] Lustre: server umount lustre-OST0000 complete [ 8161.259177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8166.518599] Lustre: server umount lustre-MDT0000 complete [ 8169.958873] LustreError: 212887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8169.981340] LustreError: 212887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 8170.564375] LustreError: 212038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848713 with bad export cookie 11811198751998708886 [ 8170.567035] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8170.582685] LustreError: 212038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8170.937812] Lustre: server umount lustre-MDT0001 complete [ 8174.771102] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 12:38:35 (1787848715) [ 8191.391636] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 8203.224035] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 8246.414547] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.695420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8246.742736] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8246.746712] Lustre: Skipped 1 previous similar message [ 8246.825771] Lustre: lustre-MDT0000: new disk, initializing [ 8246.827993] Lustre: Skipped 1 previous similar message [ 8247.008919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8247.020297] Lustre: Skipped 1 previous similar message [ 8252.641720] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8265.026160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8265.160536] Lustre: 218629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8265.458548] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8270.095908] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8280.036362] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8285.961747] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8292.792848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8293.094176] Lustre: 220600:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8295.153196] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8299.054951] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8310.595306] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8317.316910] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8317.640526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8323.907858] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8324.377872] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8329.188416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8329.210239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8329.216519] Lustre: Skipped 3 previous similar messages [ 8329.232097] Lustre: Skipped 8 previous similar messages [ 8333.969340] Lustre: server umount lustre-OST0000 complete [ 8344.355883] Lustre: server umount lustre-MDT0000 complete [ 8344.554482] LustreError: 218641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8344.588205] LustreError: 218641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8348.388122] LustreError: 220599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848891 with bad export cookie 11811198751998709775 [ 8348.404093] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8348.701729] Lustre: server umount lustre-MDT0001 complete [ 8358.600327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.167157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8359.173178] Lustre: Skipped 5 previous similar messages [ 8364.741782] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8374.588425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.486606] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8388.164317] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8394.860180] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8403.395333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8410.360883] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8418.352407] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8430.567463] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8430.586789] LustreError: Skipped 1 previous similar message [ 8434.903290] Lustre: server umount lustre-OST0000 complete [ 8444.778070] Lustre: server umount lustre-MDT0000 complete [ 8448.443574] LustreError: 222798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787848991 with bad export cookie 11811198751998710657 [ 8448.455784] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8448.457831] LustreError: 222798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 8448.788725] Lustre: server umount lustre-MDT0001 complete [ 8558.773412] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 8561.416696] Key type lgssc unregistered [ 8561.765621] LNet: 226483:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8561.771357] LNetError: 226483:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8561.804739] LNet: Removed LNI 192.168.206.124@tcp [ 8562.774154] Key type .llcrypt unregistered [ 8562.779086] Key type ._llcrypt unregistered [ 8591.963847] Key type ._llcrypt registered [ 8591.972160] Key type .llcrypt registered [ 8592.084174] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 8607.245252] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 8607.998591] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8608.051769] alg: No test for adler32 (adler32-zlib) [ 8609.237855] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 8609.485315] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 8611.151622] Key type lgssc registered [ 8612.036942] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8648.462702] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8648.514770] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8649.845469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8649.871833] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8649.961763] Lustre: lustre-MDT0000: new disk, initializing [ 8650.041112] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8650.060697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8654.720927] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8668.767499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8668.960940] Lustre: 229840:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8669.048788] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8669.055364] Lustre: Skipped 1 previous similar message [ 8669.196106] Lustre: lustre-MDT0001: new disk, initializing [ 8669.358952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8669.431501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8669.457690] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8675.020295] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8683.905788] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8689.002572] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8695.607589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8695.885743] Lustre: lustre-OST0000: new disk, initializing [ 8695.889702] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8695.897908] Lustre: 231807:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8695.973490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8701.627306] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8704.136644] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8704.149758] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8704.292453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8711.175083] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8716.815375] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8717.134516] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8723.695976] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8724.177321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8730.084505] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8730.092553] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8730.102682] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8730.117886] Lustre: Skipped 1 previous similar message [ 8730.135490] Lustre: Skipped 1 previous similar message [ 8733.853126] Lustre: server umount lustre-OST0000 complete [ 8740.327749] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8740.341270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8740.354142] Lustre: Skipped 1 previous similar message [ 8740.372109] Lustre: Skipped 1 previous similar message [ 8745.444061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8745.455662] Lustre: Skipped 1 previous similar message [ 8745.647574] Lustre: server umount lustre-MDT0000 complete [ 8749.058859] LustreError: 229831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849291 with bad export cookie 17521997830337345554 [ 8749.067234] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8749.069228] LustreError: 229831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8749.408976] Lustre: server umount lustre-MDT0001 complete [ 8767.698655] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 8779.061053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8779.465942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8784.213910] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8784.865290] LustreError: 234740:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8784.907958] LustreError: 234740:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8789.986973] LustreError: 234741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8792.423925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8792.842948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8797.432718] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8800.744669] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8808.424643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8809.012898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.967807] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8823.561816] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8823.746980] Lustre: lustre-OST0001: new disk, initializing [ 8823.750552] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8823.756481] Lustre: 237049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8830.581694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8830.591353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8830.672729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8830.729248] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8841.867472] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8847.865652] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8860.649630] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 12:50:01 (1787849401) [ 8864.737208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8864.744946] LustreError: Skipped 1 previous similar message [ 8864.747348] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8864.764763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8866.791756] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8866.805137] Lustre: Skipped 2 previous similar messages [ 8869.155850] Lustre: server umount lustre-MDT0000 complete [ 8871.913328] LustreError: 237391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8871.941205] LustreError: 237391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8872.390414] LustreError: 235474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849415 with bad export cookie 17521997830337346436 [ 8872.401429] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8872.409225] LustreError: 235474:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8872.822614] Lustre: server umount lustre-MDT0001 complete [ 8886.438012] Lustre: server umount lustre-OST0000 complete [ 8899.564204] Lustre: server umount lustre-OST0001 complete [ 8906.905232] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 8917.967459] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 8959.686032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8960.127269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8960.186960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8960.260610] Lustre: lustre-MDT0000: new disk, initializing [ 8960.439242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8960.454987] Lustre: Skipped 1 previous similar message [ 8960.481435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8965.897671] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 8984.301902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8984.515272] Lustre: 241000:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8984.594988] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8984.598600] Lustre: Skipped 1 previous similar message [ 8984.670408] Lustre: lustre-MDT0001: new disk, initializing [ 8984.820454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8984.832031] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8990.270819] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9002.176206] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9009.689670] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9018.513837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9018.990351] Lustre: lustre-OST0000: new disk, initializing [ 9019.000267] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9019.015822] Lustre: 242972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9019.156252] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9019.170716] Lustre: Skipped 1 previous similar message [ 9020.726110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9020.750993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9021.018261] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9027.335170] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9038.154976] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9043.236397] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9043.594830] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9048.461160] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9048.641176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9051.624821] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9051.628519] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9051.647987] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.652331] Lustre: Skipped 1 previous similar message [ 9051.670694] Lustre: Skipped 5 previous similar messages [ 9056.690944] Lustre: server umount lustre-OST0000 complete [ 9066.980182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9066.988442] LustreError: Skipped 1 previous similar message [ 9066.989831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9067.008896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9067.014933] Lustre: Skipped 1 previous similar message [ 9067.034349] Lustre: Skipped 1 previous similar message [ 9068.961966] Lustre: server umount lustre-MDT0000 complete [ 9072.096360] LustreError: 242234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9072.111936] LustreError: 242234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9072.660437] LustreError: 240991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849615 with bad export cookie 17521997830337348571 [ 9072.665028] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9072.669516] LustreError: 240991:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9072.947290] Lustre: server umount lustre-MDT0001 complete [ 9081.202198] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9081.692351] LustreError: 245187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9086.260887] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9092.075382] LustreError: 245187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.099707] LustreError: 245187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9094.360486] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9094.734704] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9094.742440] Lustre: Skipped 1 previous similar message [ 9098.784204] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9106.302813] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9111.413267] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9117.592671] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9123.847481] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9131.336784] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9278.946113] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9278.964460] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9278.984481] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9285.078387] Lustre: server umount lustre-OST0000 complete [ 9289.199808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9289.213809] LustreError: Skipped 1 previous similar message [ 9295.537557] Lustre: server umount lustre-MDT0000 complete [ 9296.354188] LustreError: 245188:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9296.373682] LustreError: 245188:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9299.296382] LustreError: 245168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787849842 with bad export cookie 17521997830337349453 [ 9299.308542] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9299.621482] Lustre: server umount lustre-MDT0001 complete [ 9399.359736] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [ 9402.297627] Key type lgssc unregistered [ 9402.558904] LNet: 248796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9402.574634] LNetError: 248796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9402.601075] LNet: Removed LNI 192.168.206.124@tcp [ 9403.706176] Key type .llcrypt unregistered [ 9403.711786] Key type ._llcrypt unregistered [ 9429.560842] Key type ._llcrypt registered [ 9429.564213] Key type .llcrypt registered [ 9429.672899] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 9444.803880] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 9445.956927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9445.991609] alg: No test for adler32 (adler32-zlib) [ 9447.199469] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [ 9447.405270] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [ 9449.095162] Key type lgssc registered [ 9450.415518] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9496.548525] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 9505.675960] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9505.720227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9507.105163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9507.155741] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9507.319840] Lustre: lustre-MDT0000: new disk, initializing [ 9507.464912] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9507.506939] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9512.392261] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9523.320573] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9523.590262] Lustre: 252869:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9523.698607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9523.713556] Lustre: Skipped 1 previous similar message [ 9523.852611] Lustre: lustre-MDT0001: new disk, initializing [ 9523.980470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9524.035443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9524.047876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9528.781643] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9533.595734] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9540.867647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.245429] Lustre: lustre-OST0000: new disk, initializing [ 9541.252565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9541.272313] Lustre: 254511:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9541.390836] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9548.096840] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9550.959486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9550.983558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9551.100456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9558.749120] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9558.908317] Lustre: lustre-OST0001: new disk, initializing [ 9558.911839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9558.917637] Lustre: 255383:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9559.036712] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9560.298521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9560.308564] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9560.410645] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9564.351766] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9575.211121] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9581.454719] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9595.878710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9595.887551] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9595.906609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9596.392540] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9596.403308] Lustre: Skipped 2 previous similar messages [ 9596.415995] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9596.424030] Lustre: Skipped 3 previous similar messages [ 9601.511770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9601.535480] Lustre: Skipped 3 previous similar messages [ 9606.111461] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9606.296067] Lustre: server umount lustre-MDT0000 complete [ 9610.441751] LustreError: 252861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850153 with bad export cookie 1849788128722585964 [ 9610.449908] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9610.453328] LustreError: 252861:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9610.736600] Lustre: server umount lustre-MDT0001 complete [ 9624.949375] Lustre: server umount lustre-OST0000 complete [ 9628.128030] Lustre: 250376:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787850154/real 1787850154] req@ffff99ada0773c00 x1874696591194752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787850170 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9628.151558] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9629.244336] Lustre: server umount lustre-OST0001 complete [ 9645.424279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9655.293617] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9664.280713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9675.604180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9697.076410] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 9707.499000] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9707.520899] Lustre: lustre-MDT0000: reset Object Index mappings [ 9707.998819] LustreError: 260394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9708.016418] LustreError: 260394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 9708.060996] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9712.231759] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9713.127834] LustreError: 260394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9713.142127] LustreError: 260394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9718.266232] LustreError: 260395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9720.083779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9720.098684] Lustre: lustre-MDT0001: reset Object Index mappings [ 9720.567818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9724.597776] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9727.387562] Lustre: 261535:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9734.059723] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9740.499278] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9745.715598] LustreError: 261891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9748.992888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9749.242966] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9749.250843] Lustre: Skipped 1 previous similar message [ 9753.342117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9753.349612] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9755.779669] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9762.756593] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9766.292636] Lustre: 263408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9772.004134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9772.015263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9772.032768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9772.686741] Lustre: server umount lustre-MDT0000 complete [ 9774.059810] LustreError: 261127:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9774.081000] LustreError: 261127:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9775.682498] LustreError: 260375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850318 with bad export cookie 1849788128722592754 [ 9775.691420] LustreError: 260375:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9775.693358] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9776.056466] Lustre: server umount lustre-MDT0001 complete [ 9789.755148] Lustre: server umount lustre-OST0000 complete [ 9802.587583] Lustre: server umount lustre-OST0001 complete [ 9823.626869] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 13:06:04 (1787850364) [ 9840.067312] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [ 9849.567837] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 9882.275695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9882.478645] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9882.506559] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9882.578583] Lustre: lustre-MDT0000: new disk, initializing [ 9882.694492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9882.708423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9887.324793] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9896.879626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9896.998839] Lustre: 267663:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9897.040629] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9897.044388] Lustre: Skipped 1 previous similar message [ 9897.114397] Lustre: lustre-MDT0001: new disk, initializing [ 9897.232088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9897.248365] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9901.034652] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9908.770221] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9913.953921] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9921.043855] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9921.349738] Lustre: lustre-OST0000: new disk, initializing [ 9921.355082] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9921.367521] Lustre: 269629:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9922.818176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9922.830428] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9922.939429] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9927.147206] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [ 9935.674040] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9940.068671] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9940.320248] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9944.865906] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9945.147703] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9948.669363] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9948.677664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9948.683433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9948.696300] Lustre: Skipped 4 previous similar messages [ 9952.998907] Lustre: server umount lustre-OST0000 complete [ 9958.882172] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9958.903447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9958.907567] Lustre: Skipped 1 previous similar message [ 9963.439279] Lustre: server umount lustre-MDT0000 complete [ 9964.000947] LustreError: 268545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9964.019926] LustreError: 268545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9967.415292] LustreError: 267654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850510 with bad export cookie 1849788128722594847 [ 9967.418722] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9967.431180] LustreError: 267654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9967.775965] Lustre: server umount lustre-MDT0001 complete [ 9986.380482] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [ 9996.726784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9997.074795] LustreError: 272563:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9997.142512] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9997.148459] Lustre: Skipped 2 previous similar messages [10001.770478] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10011.782992] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10016.522393] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10019.326895] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10026.991163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10034.885285] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10042.538912] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10042.686726] Lustre: lustre-OST0001: new disk, initializing [10042.693035] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10042.700525] Lustre: 274872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10043.894826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10043.909813] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10044.060541] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10049.174871] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10059.499332] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10063.177318] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10068.864327] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10068.953320] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10078.696206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10078.713709] LustreError: Skipped 1 previous similar message [10078.717966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10078.732210] Lustre: Skipped 1 previous similar message [10078.735247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10081.956313] Lustre: server umount lustre-MDT0000 complete [10084.837706] LustreError: 273588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10084.853719] LustreError: 273588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10085.800825] LustreError: 274871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850628 with bad export cookie 1849788128722595729 [10085.808525] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10085.811219] LustreError: 274871:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10086.289809] Lustre: server umount lustre-MDT0001 complete [10100.690866] Lustre: server umount lustre-OST0000 complete [10115.261478] Lustre: server umount lustre-OST0001 complete [10132.584592] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [10143.089532] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.496970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10143.502645] Lustre: Skipped 3 previous similar messages [10148.099186] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10148.839885] LustreError: 277930:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10148.861982] LustreError: 277930:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10157.054581] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10162.225819] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10165.503167] Lustre: 279071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10172.475678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10179.845695] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10189.331238] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10190.043888] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10195.456867] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10197.984419] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10205.519040] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10209.074583] Lustre: 280947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10214.229203] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10214.235063] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10223.762852] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 13:12:44 (1787850764) [10239.456622] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10239.464315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10239.483563] Lustre: Skipped 3 previous similar messages [10239.508282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10239.514846] Lustre: Skipped 4 previous similar messages [10245.236871] Lustre: server umount lustre-MDT0000 complete [10249.201723] LustreError: 277910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787850791 with bad export cookie 1849788128722602484 [10249.211358] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10249.213092] LustreError: 277910:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10255.807455] Lustre: server umount lustre-MDT0001 complete [10265.762771] Lustre: server umount lustre-OST0000 complete [10275.309237] Lustre: server umount lustre-OST0001 complete [10281.964703] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [10294.855246] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [10351.877086] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [10362.105419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10362.337824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10362.371926] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10362.472417] Lustre: lustre-MDT0000: new disk, initializing [10362.544575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10367.486984] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10378.155372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10378.281413] Lustre: 285191:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10378.613731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10383.952813] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10396.724325] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10397.068321] Lustre: 286825:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10399.186976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10403.216657] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10413.249213] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10413.473833] Lustre: 287695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10413.578901] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10413.585960] Lustre: Skipped 6 previous similar messages [10415.023394] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10415.030395] Lustre: Skipped 1 previous similar message [10415.141056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10419.644865] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10428.852969] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10432.592878] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10432.599488] Lustre: Skipped 1 previous similar message [10447.014150] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 13:16:27 (1787850987) [10450.400535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10450.412261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10450.427344] Lustre: Skipped 5 previous similar messages [10450.433884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10450.438279] Lustre: Skipped 8 previous similar messages [10455.089922] Lustre: server umount lustre-MDT0000 complete [10456.037459] LustreError: 285203:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10456.055885] LustreError: 285203:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 16 previous similar messages [10457.852805] LustreError: 285183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851000 with bad export cookie 1849788128722609904 [10457.858891] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10457.861872] LustreError: 285183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10458.112600] Lustre: server umount lustre-MDT0001 complete [10470.441742] Lustre: server umount lustre-OST0000 complete [10484.056698] Lustre: server umount lustre-OST0001 complete [10491.676970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10496.311494] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10504.299644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10509.299930] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10518.039205] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10523.835787] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10530.261549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10535.807420] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10542.235390] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10566.324247] loop1: detected capacity change from 0 to 204800000 [10566.488248] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10566.498749] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10566.520969] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10571.447878] loop1: detected capacity change from 0 to 204800000 [10576.797915] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10576.927388] LustreError: 289931:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs29152: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10576.939614] LustreError: 289931:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs29152 [10576.944622] LustreError: 289931:0:(mgs_handler.c:517:mgs_target_reg()) fs29152-MDT0001: Error checking target index: rc = -13 [10576.953830] LustreError: fs29152-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10576.966699] LustreError: 293683:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10576.975817] LustreError: 293683:0:(tgt_mount.c:2129:server_put_super()) no obd fs29152-MDT0001 [10576.983817] LustreError: 293683:0:(tgt_mount.c:116:server_deregister_mount()) fs29152-MDT0001 not registered [10577.055167] Lustre: server umount fs29152-MDT0001 complete [10577.062321] LustreError: 293683:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10584.768781] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10584.878533] Lustre: Setting parameter fs29152-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs29152-MDT0001 [10584.927244] Lustre: srv-fs29152-MDT0001: No data found on store. Initialize space. [10584.932255] Lustre: Skipped 4 previous similar messages [10585.029961] Lustre: fs29152-MDT0001: new disk, initializing [10585.033874] Lustre: Skipped 3 previous similar messages [10585.122813] Lustre: srv-fs29152-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10585.131416] Lustre: 294149:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29152-MDT0001-mdtlov: can't create new llog: rc = -115 [10585.139587] LustreError: 294149:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29152-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10589.628729] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10596.127826] Lustre: server umount fs29152-MDT0001 complete [10608.623608] loop1: detected capacity change from 0 to 204800000 [10613.343696] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10613.627857] Lustre: 295716:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29152-MDT0001-mdtlov: can't create new llog: rc = -115 [10613.640372] LustreError: 295716:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29152-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10616.807551] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10617.092061] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10620.735671] Lustre: server umount fs29152-MDT0001 complete [10629.629783] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.24@tcp (stopping) [10629.637136] Lustre: Skipped 3 previous similar messages [10632.166855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10632.176083] Lustre: Skipped 5 previous similar messages [10634.794927] Lustre: server umount lustre-MDT0000 complete [10641.279406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10641.385723] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10645.742151] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10647.020743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10651.763099] loop1: detected capacity change from 0 to 204800000 [10655.960245] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10656.305489] Lustre: 298161:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs29152-MDT0001-mdtlov: can't create new llog: rc = -115 [10656.318113] LustreError: 298161:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs29152-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10659.740878] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10663.633848] Lustre: server umount fs29152-MDT0001 complete [10682.861317] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10682.867318] LustreError: Skipped 1 previous similar message [10696.159624] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10696.322237] Lustre: server umount lustre-OST0000 complete [10706.647523] Lustre: server umount lustre-MDT0000 complete [10711.119169] LustreError: 289923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851253 with bad export cookie 1849788128722612585 [10711.127656] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10711.133436] LustreError: 289923:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10711.743760] Lustre: server umount lustre-MDT0001 complete [10726.409257] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [10730.176288] Key type lgssc unregistered [10730.532182] LNet: 300512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10730.542182] LNetError: 300512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10730.566320] LNet: Removed LNI 192.168.206.124@tcp [10731.630569] Key type .llcrypt unregistered [10731.634435] Key type ._llcrypt unregistered [10753.239258] Key type ._llcrypt registered [10753.242863] Key type .llcrypt registered [10753.357107] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [10754.628151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10754.866643] alg: No test for adler32 (adler32-zlib) [10755.915176] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [10756.168298] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [10757.856804] Key type lgssc registered [10759.309106] Lustre: Echo OBD driver; http://www.lustre.org/ [10767.895965] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10767.912156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10769.297103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10774.279884] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10779.619444] LustreError: 301784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.667602] LustreError: 301784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10782.367742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10782.879857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10787.259423] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10794.999537] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10800.581934] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10806.917795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10807.272898] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10808.307945] LustreError: 303611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.419819] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10813.441095] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10819.567034] LustreError: 303614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10819.607878] LustreError: 303614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10820.281449] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10824.672646] LustreError: 303696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10824.696546] LustreError: 303696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10833.406922] LustreError: 303611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10833.424509] LustreError: 303611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10837.197368] loop1: detected capacity change from 0 to 204800000 [10837.373535] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10837.386810] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10837.396629] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10841.893602] loop1: detected capacity change from 0 to 204800000 [10847.215142] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10847.407549] Lustre: fs20037-OST0000: new disk, initializing [10847.411778] Lustre: srv-fs20037-OST0000: No data found on store. Initialize space. [10847.422441] Lustre: 305517:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10847.483405] Lustre: fs20037-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10850.276064] LustreError: 303696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.298022] LustreError: 303696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [10853.953827] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10862.353822] Lustre: server umount fs20037-OST0000 complete [10873.926906] loop1: detected capacity change from 0 to 204800000 [10879.768987] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10880.034355] Lustre: fs20037-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10884.588686] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10884.611254] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 18 previous similar messages [10885.790754] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10891.244783] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10894.565064] loop2: detected capacity change from 0 to 204800000 [10894.814868] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10894.847857] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10894.860487] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10896.363059] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10900.575924] loop2: detected capacity change from 0 to 204800000 [10906.566987] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10906.732140] LustreError: fs20037-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10906.746299] LustreError: fs20037-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10906.764924] LustreError: 308562:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -13 [10906.773206] LustreError: 308562:0:(tgt_mount.c:2129:server_put_super()) no obd fs20037-OST0001 [10906.780340] LustreError: 308562:0:(tgt_mount.c:116:server_deregister_mount()) fs20037-OST0001 not registered [10906.830984] Lustre: server umount fs20037-OST0001 complete [10906.834731] LustreError: 308562:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10917.708952] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10918.007264] Lustre: fs20037-OST0001: new disk, initializing [10918.017811] Lustre: srv-fs20037-OST0001: No data found on store. Initialize space. [10918.026839] Lustre: 309115:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10918.138248] Lustre: fs20037-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10924.181230] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10932.939879] Lustre: server umount fs20037-OST0000 complete [10943.027178] Lustre: server umount fs20037-OST0001 complete [10951.158376] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10951.173809] LustreError: 303613:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [10956.539854] loop1: detected capacity change from 0 to 204800000 [10962.470487] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10962.738732] Lustre: fs20037-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10969.590654] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [10978.132880] loop2: detected capacity change from 0 to 204800000 [10984.891149] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10993.398785] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11004.218132] Lustre: server umount fs20037-OST0000 complete [11013.951782] Lustre: server umount fs20037-OST0001 complete [11028.449646] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11028.469539] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11028.507475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11029.474417] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11029.486078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11029.507518] Lustre: Skipped 1 previous similar message [11033.549416] Lustre: server umount lustre-OST0000 complete [11038.688439] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11038.693568] LustreError: Skipped 1 previous similar message [11038.696312] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11038.706147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11043.644259] Lustre: server umount lustre-MDT0000 complete [11047.383662] LustreError: 309114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851590 with bad export cookie 420445402104166381 [11047.389440] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11047.400812] LustreError: 309114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11047.791432] Lustre: server umount lustre-MDT0001 complete [11059.814679] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [11063.611739] Key type lgssc unregistered [11063.966366] LNet: 315041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11063.989996] LNetError: 315041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11064.034865] LNet: Removed LNI 192.168.206.124@tcp [11065.107351] Key type .llcrypt unregistered [11065.114128] Key type ._llcrypt unregistered [11096.803873] Key type ._llcrypt registered [11096.808068] Key type .llcrypt registered [11096.896833] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [11113.016877] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [11114.716775] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11114.892979] alg: No test for adler32 (adler32-zlib) [11116.008910] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11116.360690] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [11118.135357] Key type lgssc registered [11119.375641] Lustre: Echo OBD driver; http://www.lustre.org/ [11167.960983] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11167.977487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11169.295979] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11169.346425] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11169.471808] Lustre: lustre-MDT0000: new disk, initializing [11169.617457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11169.697907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11176.455404] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11192.862494] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11193.068588] Lustre: 318905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11193.167425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11193.178138] Lustre: Skipped 1 previous similar message [11193.266637] Lustre: lustre-MDT0001: new disk, initializing [11193.371821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11193.437782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11193.469339] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11199.798711] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11209.604341] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11215.787694] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11222.585505] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11222.909535] Lustre: lustre-OST0000: new disk, initializing [11222.916304] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11222.928289] Lustre: 320872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11223.055964] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11228.278198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11228.290935] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11228.426338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11230.449266] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11240.572275] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11243.245026] Lustre: Mounted lustre-client - version 2.17.57_65_g89f8e4a [11244.339948] Lustre: Unmounted lustre-client [11258.857440] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11258.860599] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11258.880865] Lustre: Skipped 1 previous similar message [11260.812986] Lustre: server umount lustre-OST0000 complete [11264.996239] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11265.002499] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.016903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11265.022493] Lustre: Skipped 1 previous similar message [11269.090712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11269.092907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11274.210703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11274.223186] Lustre: Skipped 2 previous similar messages [11278.308776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11278.313160] Lustre: Skipped 1 previous similar message [11279.327153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11279.566593] Lustre: server umount lustre-MDT0000 complete [11283.590486] LustreError: 318897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787851826 with bad export cookie 15258447517879034427 [11283.599826] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11283.604618] LustreError: 318897:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11284.030577] Lustre: server umount lustre-MDT0001 complete [11295.190522] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [11298.309587] Key type lgssc unregistered [11299.679229] LNet: 1 local NIs in recovery (showing 1): 192.168.206.124@tcp [11299.689524] LNet: 323036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11299.713747] LNetError: 323036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11372.454136] LNet: Removed LNI 192.168.206.124@tcp [11373.401497] Key type .llcrypt unregistered [11373.407383] Key type ._llcrypt unregistered [11392.787345] Key type ._llcrypt registered [11392.793902] Key type .llcrypt registered [11392.921163] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [11410.049422] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [11410.714383] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11410.862662] alg: No test for adler32 (adler32-zlib) [11412.159412] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11412.457398] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [11414.183421] Key type lgssc registered [11415.230216] Lustre: Echo OBD driver; http://www.lustre.org/ [11451.664761] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 13:33:11 (1787851991) [11459.069256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11459.099675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11460.451736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11460.506763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11460.660183] Lustre: lustre-MDT0000: new disk, initializing [11460.727163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11460.745969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11466.175852] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11477.394452] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11477.556662] Lustre: 326257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11477.606255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11477.617727] Lustre: Skipped 1 previous similar message [11477.745980] Lustre: lustre-MDT0001: new disk, initializing [11477.910939] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11477.970975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11477.983201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11483.006026] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11495.499390] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11501.939325] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11509.620926] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11509.902534] Lustre: lustre-OST0000: new disk, initializing [11509.905933] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11509.913129] Lustre: 328226:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11510.056092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11515.940752] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11516.984051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11517.001684] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11517.159152] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11525.468669] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11543.011649] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11543.015484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11543.019941] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11543.027405] LustreError: Skipped 1 previous similar message [11543.068650] Lustre: Skipped 1 previous similar message [11544.907446] Lustre: server umount lustre-OST0000 complete [11549.666018] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11549.677941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11549.712242] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11549.717159] Lustre: Skipped 1 previous similar message [11553.259484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11553.271753] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11554.643043] Lustre: server umount lustre-MDT0000 complete [11559.517931] LustreError: 326250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852102 with bad export cookie 81255525797259827 [11559.533682] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11559.538132] LustreError: 326250:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11559.869971] Lustre: server umount lustre-MDT0001 complete [11574.660735] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [11578.240873] Key type lgssc unregistered [11578.644678] LNet: 330142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11578.654770] LNetError: 330142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11578.677910] LNet: Removed LNI 192.168.206.124@tcp [11579.872498] Key type .llcrypt unregistered [11579.874228] Key type ._llcrypt unregistered [11606.064672] Key type ._llcrypt registered [11606.066685] Key type .llcrypt registered [11606.196945] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [11607.037675] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11607.100106] alg: No test for adler32 (adler32-zlib) [11608.339310] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11608.627486] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [11610.407175] Key type lgssc registered [11612.393536] Lustre: Echo OBD driver; http://www.lustre.org/ [11624.259181] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11624.312443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11626.117455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11632.183703] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11635.681567] LustreError: 331417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11635.697854] LustreError: 331417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11640.825972] LustreError: 331416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11643.900974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11644.427075] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11649.297089] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11657.445542] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11663.597893] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11671.157066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11671.572029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11677.943716] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11686.274961] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11708.606806] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11710.289539] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11716.064846] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11716.076250] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11716.109707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11716.576390] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11716.595916] Lustre: Skipped 1 previous similar message [11718.671569] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.24@tcp (stopping) [11718.678806] Lustre: Skipped 2 previous similar messages [11721.710223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11721.721750] Lustre: Skipped 2 previous similar messages [11723.757836] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.24@tcp (stopping) [11728.352258] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11728.537652] Lustre: server umount lustre-MDT0000 complete [11728.877646] LustreError: 331411:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.24@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.937446] LustreError: 331416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.964173] LustreError: 331416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11733.200245] LustreError: 331398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852275 with bad export cookie 8795949282426069324 [11733.201426] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11733.222354] LustreError: 331398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11734.018562] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.24@tcp (stopping) [11734.024945] Lustre: Skipped 3 previous similar messages [11737.062628] LustreError: 331412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11737.067343] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11737.089343] LustreError: 331412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11739.594482] Lustre: server umount lustre-MDT0001 complete [11743.837210] Lustre: server umount lustre-OST0000 complete [11752.514443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.551606] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11760.496652] Lustre: 335621:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.124@tcp [11762.608810] Lustre: server umount MGS complete [11771.915993] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11772.521463] LustreError: 336194:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11772.561285] LustreError: 336194:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11772.619174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11777.981676] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11787.597357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11788.066186] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11792.892078] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11801.276801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11801.824784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11808.511780] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11815.976259] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11817.716274] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11824.214705] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11827.114882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11832.807445] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [11832.818816] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11832.836292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11832.839158] Lustre: Skipped 3 previous similar messages [11833.828295] LustreError: 337631:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.832564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11833.857671] LustreError: 337631:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11833.931224] Lustre: server umount lustre-OST0000 complete [11838.945181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11838.974704] Lustre: Skipped 2 previous similar messages [11845.110974] Lustre: server umount lustre-MDT0000 complete [11850.055787] LustreError: 336176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852392 with bad export cookie 8795949282426070164 [11850.058596] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11850.082925] LustreError: 336176:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11850.886637] Lustre: server umount lustre-MDT0001 complete [11864.205818] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [11867.266780] Key type lgssc unregistered [11867.622714] LNet: 339633:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11867.641688] LNetError: 339633:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11867.681343] LNet: Removed LNI 192.168.206.124@tcp [11869.028576] Key type .llcrypt unregistered [11869.036066] Key type ._llcrypt unregistered [11891.839488] Key type ._llcrypt registered [11891.844060] Key type .llcrypt registered [11892.023584] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [11910.176212] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [11911.733413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11911.770025] alg: No test for adler32 (adler32-zlib) [11912.935792] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [11913.195449] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [11914.807731] Key type lgssc registered [11915.910975] Lustre: Echo OBD driver; http://www.lustre.org/ [11955.084221] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11955.095036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11956.279906] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11956.319151] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11956.422324] Lustre: lustre-MDT0000: new disk, initializing [11956.498101] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11956.514801] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11961.355884] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11977.515591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11977.730747] Lustre: 342810:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11977.836348] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11977.845457] Lustre: Skipped 1 previous similar message [11978.064416] Lustre: lustre-MDT0001: new disk, initializing [11978.296458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11978.407161] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11978.431941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11983.909442] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [11995.497420] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12001.819835] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12009.466487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12009.703618] Lustre: lustre-OST0000: new disk, initializing [12009.707362] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12009.712660] Lustre: 344778:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12009.794770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12017.838353] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [12018.286068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12018.299329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12018.442634] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12031.538620] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12038.926151] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12039.241062] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12043.978183] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12044.173661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12049.390163] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12049.390289] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12049.411915] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12049.419056] Lustre: Skipped 1 previous similar message [12052.413274] Lustre: server umount lustre-OST0000 complete [12059.630420] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12059.640442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12059.658263] Lustre: Skipped 2 previous similar messages [12059.674328] Lustre: Skipped 1 previous similar message [12064.736562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12064.750360] Lustre: Skipped 1 previous similar message [12069.859575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12069.869409] Lustre: Skipped 1 previous similar message [12070.367190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12070.644366] Lustre: server umount lustre-MDT0000 complete [12074.726679] LustreError: 342800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852617 with bad export cookie 12103387036580315239 [12074.753167] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12074.759188] LustreError: 342800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12075.223513] Lustre: server umount lustre-MDT0001 complete [12086.564591] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 13:43:46 (1787852626) [12110.013171] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [12120.405847] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [12165.038169] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12165.234124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12165.252300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12165.310178] Lustre: lustre-MDT0000: new disk, initializing [12165.357887] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12165.371988] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12169.562303] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [12181.261108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12181.384148] Lustre: 349544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12181.444884] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12181.447823] Lustre: Skipped 1 previous similar message [12181.539734] Lustre: lustre-MDT0001: new disk, initializing [12181.654654] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12181.717809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12181.736923] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12185.829270] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [12195.730809] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12201.427958] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12207.806942] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12208.078269] Lustre: lustre-OST0000: new disk, initializing [12208.082083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12208.090696] Lustre: 351512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12208.188761] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12213.294610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12213.304718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12213.383945] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12213.562205] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [12223.292956] Lustre: DEBUG MARKER: oleg624-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12259.828976] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12259.829809] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12259.837608] LustreError: Skipped 2 previous similar messages [12259.839691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12259.853050] Lustre: Skipped 1 previous similar message [12262.557346] Lustre: server umount lustre-OST0000 complete [12268.513082] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12268.523202] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12268.543292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12268.554490] Lustre: Skipped 1 previous similar message [12272.872076] Lustre: server umount lustre-MDT0000 complete [12277.147849] LustreError: 349537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787852819 with bad export cookie 12103387036580316121 [12277.156444] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12277.164943] LustreError: 349537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12277.798585] Lustre: server umount lustre-MDT0001 complete [12290.193844] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12293.801253] Key type lgssc unregistered [12294.061499] LNet: 353679:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12294.069639] LNetError: 353679:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12294.080751] LNet: Removed LNI 192.168.206.124@tcp [12295.118374] Key type .llcrypt unregistered [12295.123484] Key type ._llcrypt unregistered [12383.604096] Key type ._llcrypt registered [12383.608109] Key type .llcrypt registered [12383.704428] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12383.982643] Key type .llcrypt unregistered [12383.987442] Key type ._llcrypt unregistered [12454.130869] Key type ._llcrypt registered [12454.134899] Key type .llcrypt registered [12454.245307] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12454.508645] Key type .llcrypt unregistered [12454.511355] Key type ._llcrypt unregistered [12483.043526] Key type ._llcrypt registered [12483.046911] Key type .llcrypt registered [12483.165587] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12483.427911] Key type .llcrypt unregistered [12483.434469] Key type ._llcrypt unregistered [12516.004345] Key type ._llcrypt registered [12516.007573] Key type .llcrypt registered [12516.256389] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12516.809655] Key type .llcrypt unregistered [12516.812493] Key type ._llcrypt unregistered [12590.881546] Key type ._llcrypt registered [12590.883966] Key type .llcrypt registered [12590.995202] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12591.401459] Key type .llcrypt unregistered [12591.405822] Key type ._llcrypt unregistered [12623.208135] Key type ._llcrypt registered [12623.211881] Key type .llcrypt registered [12623.333547] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12623.699505] Key type .llcrypt unregistered [12623.701948] Key type ._llcrypt unregistered [12652.636665] Key type ._llcrypt registered [12652.640894] Key type .llcrypt registered [12652.734986] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12652.982506] Key type .llcrypt unregistered [12652.984892] Key type ._llcrypt unregistered [12682.577819] Key type ._llcrypt registered [12682.581994] Key type .llcrypt registered [12682.730700] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12683.018463] Key type .llcrypt unregistered [12683.021420] Key type ._llcrypt unregistered [12713.355228] Key type ._llcrypt registered [12713.356768] Key type .llcrypt registered [12713.451417] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12713.726878] Key type .llcrypt unregistered [12713.729679] Key type ._llcrypt unregistered [12742.041926] Key type ._llcrypt registered [12742.044989] Key type .llcrypt registered [12742.172473] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12742.613358] Key type .llcrypt unregistered [12742.617171] Key type ._llcrypt unregistered [12810.594607] Key type ._llcrypt registered [12810.601460] Key type .llcrypt registered [12810.691480] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12810.984428] Key type .llcrypt unregistered [12810.987137] Key type ._llcrypt unregistered [12839.008343] Key type ._llcrypt registered [12839.011428] Key type .llcrypt registered [12839.126692] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12839.452410] Key type .llcrypt unregistered [12839.457863] Key type ._llcrypt unregistered [12868.497759] Key type ._llcrypt registered [12868.500240] Key type .llcrypt registered [12868.637985] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [12869.060304] Key type .llcrypt unregistered [12869.064865] Key type ._llcrypt unregistered [12919.041958] Key type ._llcrypt registered [12919.045257] Key type .llcrypt registered [12919.180555] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [12934.824623] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [12935.656784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12935.670607] alg: No test for adler32 (adler32-zlib) [12936.785159] Lustre: Lustre: Build Version: 2.17.57_65_g89f8e4a [12936.954812] LNet: Added LNI 192.168.206.124@tcp [8/256/0/180] [12938.623158] Key type lgssc registered [12939.914869] Lustre: Echo OBD driver; http://www.lustre.org/ [12977.014176] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12977.051210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12978.322269] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12978.357319] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12978.466737] Lustre: tst250fs-MDT0000: new disk, initializing [12978.582760] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12978.612918] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12982.497321] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [12994.065625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12994.178157] Lustre: 373987:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12994.224660] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12994.228516] Lustre: Skipped 1 previous similar message [12994.282236] Lustre: tst250fs-MDT0001: new disk, initializing [12994.374650] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12994.413503] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12994.425770] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12998.733698] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [13003.645182] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13010.630292] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13010.945479] Lustre: tst250fs-OST0000: new disk, initializing [13010.950271] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13010.958680] Lustre: 375620:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13011.093805] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13012.958066] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13012.972412] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13013.124439] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13016.851732] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [13026.843168] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13027.041391] Lustre: tst250fs-OST0001: new disk, initializing [13027.045578] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13027.052874] Lustre: 376494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13027.150501] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13033.617178] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_default_debug -1 all [13036.641267] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13036.655530] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13036.773177] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13038.858735] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13041.432919] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13041.447092] Lustre: Skipped 2 previous similar messages [13057.516111] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13057.520326] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13057.531899] Lustre: Skipped 2 previous similar messages [13057.541705] Lustre: Skipped 3 previous similar messages [13062.624464] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13062.629458] Lustre: Skipped 3 previous similar messages [13067.761965] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13071.327431] Lustre: tst250fs-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13071.532103] Lustre: server umount tst250fs-MDT0000 complete [13075.270233] LustreError: 373981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787853618 with bad export cookie 13951963003251366442 [13075.273045] LustreError: MGC192.168.206.124@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13075.300198] LustreError: 373981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13075.589094] Lustre: server umount tst250fs-MDT0001 complete [13089.916467] Lustre: server umount tst250fs-OST0000 complete [13103.773969] Lustre: server umount tst250fs-OST0001 complete [13122.176292] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [13130.576993] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [13162.461650] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 14:01:43 (1787853703) [13164.096804] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13180.088347] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing set_hostid [13190.026266] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing load_modules_local [13218.316739] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12991 sec ========= 14:02:38 (1787853758) [13219.724563] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:02:40 (1787853760) === [13222.484855] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:02:43 (1787853763) === [13256.940987] Lustre: DEBUG MARKER: oleg624-server.virtnet: executing unload_modules_local [13259.431244] Key type lgssc unregistered [13259.756616] LNet: 385637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13259.760529] LNetError: 385637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13259.780242] LNet: Removed LNI 192.168.206.124@tcp [13260.601204] Key type .llcrypt unregistered [13260.610360] Key type ._llcrypt unregistered