[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 532547919 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.003195] x2apic enabled [ 0.004006] Switched APIC routing to physical x2apic. [ 0.005008] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008019] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009010] pid_max: default: 32768 minimum: 301 [ 0.010122] LSM: Security Framework initializing [ 0.011040] Yama: becoming mindful. [ 0.012024] SELinux: Initializing. [ 0.013055] *** VALIDATE selinux *** [ 0.021270] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026144] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027164] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029102] *** VALIDATE tmpfs *** [ 0.031370] *** VALIDATE proc *** [ 0.032212] *** VALIDATE cgroup *** [ 0.033007] *** VALIDATE cgroup2 *** [ 0.034254] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.035126] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.036003] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037019] Spectre V2 : User space: Vulnerable [ 0.038004] Speculative Store Bypass: Vulnerable [ 0.041518] debug: unmapping init [mem 0xffffffff93a59000-0xffffffff93a60fff] [ 0.044000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044595] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045021] ... version: 2 [ 0.046008] ... bit width: 48 [ 0.047008] ... generic registers: 4 [ 0.048008] ... value mask: 0000ffffffffffff [ 0.049009] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.051008] ... event mask: 000000070000000f [ 0.053182] rcu: Hierarchical SRCU implementation. [ 0.055419] smp: Bringing up secondary CPUs ... [ 0.056486] x86: Booting SMP configuration: [ 0.057012] .... node #0, CPUs: #1 #2 #3 [ 0.070559] smp: Brought up 1 node, 4 CPUs [ 0.072013] smpboot: Max logical packages: 1 [ 0.073011] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.139019] node 0 deferred pages initialised in 64ms [ 0.144186] devtmpfs: initialized [ 0.145253] x86/mm: Memory block size: 128MB [ 0.148127] gcov: version magic: 0x41383552 [ 0.150394] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151082] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153135] pinctrl core: initialized pinctrl subsystem [ 0.154229] [ 0.154935] ************************************************************* [ 0.155016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.156016] ** ** [ 0.157015] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.158019] ** ** [ 0.159019] ** This means that this kernel is built to expose internal ** [ 0.160015] ** IOMMU data structures, which may compromise security on ** [ 0.161030] ** your system. ** [ 0.162019] ** ** [ 0.163010] ** If you see this message and you are not debugging the ** [ 0.164017] ** kernel, report this immediately to your vendor! ** [ 0.165012] ** ** [ 0.166009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.167013] ************************************************************* [ 0.168688] NET: Registered protocol family 16 [ 0.169000] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.180010] cpuidle: using governor menu [ 0.182539] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.186775] PCI: Using configuration type 1 for base access [ 0.189144] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.200140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.202061] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.207177] cryptd: max_cpu_qlen set to 1000 [ 0.210415] ACPI: Added _OSI(Module Device) [ 0.213014] ACPI: Added _OSI(Processor Device) [ 0.214014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.217015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.223245] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.233438] ACPI: Interpreter enabled [ 0.235076] ACPI: PM: (supports S0 S3 S4 S5) [ 0.237103] ACPI: Using IOAPIC for interrupt routing [ 0.238317] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.243699] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.258053] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.261048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.263016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.268183] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.275806] acpiphp: Slot [2] registered [ 0.277179] acpiphp: Slot [5] registered [ 0.279138] acpiphp: Slot [6] registered [ 0.281166] acpiphp: Slot [7] registered [ 0.283124] acpiphp: Slot [8] registered [ 0.285123] acpiphp: Slot [9] registered [ 0.287114] acpiphp: Slot [10] registered [ 0.289132] acpiphp: Slot [3] registered [ 0.291127] acpiphp: Slot [4] registered [ 0.293339] acpiphp: Slot [11] registered [ 0.295076] acpiphp: Slot [12] registered [ 0.296127] acpiphp: Slot [13] registered [ 0.297083] acpiphp: Slot [14] registered [ 0.298114] acpiphp: Slot [15] registered [ 0.300087] acpiphp: Slot [16] registered [ 0.301090] acpiphp: Slot [17] registered [ 0.302181] acpiphp: Slot [18] registered [ 0.303109] acpiphp: Slot [19] registered [ 0.304000] acpiphp: Slot [20] registered [ 0.304000] acpiphp: Slot [21] registered [ 0.304000] acpiphp: Slot [22] registered [ 0.304000] acpiphp: Slot [23] registered [ 0.304000] acpiphp: Slot [24] registered [ 0.304000] acpiphp: Slot [25] registered [ 0.304000] acpiphp: Slot [26] registered [ 0.304000] acpiphp: Slot [27] registered [ 0.307094] acpiphp: Slot [28] registered [ 0.308000] acpiphp: Slot [29] registered [ 0.310162] acpiphp: Slot [30] registered [ 0.312208] acpiphp: Slot [31] registered [ 0.314091] PCI host bridge to bus 0000:00 [ 0.316026] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.320030] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.323024] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.326031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.330046] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334063] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.337258] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.342505] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345923] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362022] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370052] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.372000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.372000] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.372928] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.376378] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.381066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388874] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397018] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.411015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.416013] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.421420] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.428017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.433016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.449020] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.459397] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.467016] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.474020] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.518011] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.551452] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.567066] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.596021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.648015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.676000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.693040] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.704025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.723017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.733942] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.746060] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.752016] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.771018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.784238] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.796013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.812022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.844015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.864000] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.866463] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.869476] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.871458] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.874227] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.880061] iommu: Default domain type: Passthrough [ 0.881423] SCSI subsystem initialized [ 0.882126] ACPI: bus type USB registered [ 0.883250] usbcore: registered new interface driver usbfs [ 0.886200] usbcore: registered new interface driver hub [ 0.889151] usbcore: registered new device driver usb [ 0.891246] pps_core: LinuxPPS API ver. 1 registered [ 0.894011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.897099] PTP clock support registered [ 0.900068] EDAC MC: Ver: 3.0.0 [ 0.902081] PCI: Using ACPI for IRQ routing [ 0.904227] NetLabel: Initializing [ 0.906010] NetLabel: domain hash size = 128 [ 0.908010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.911125] NetLabel: unlabeled traffic allowed by default [ 0.914447] vgaarb: loaded [ 0.916892] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.919009] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.928252] clocksource: Switched to clocksource kvm-clock [ 1.098423] VFS: Disk quotas dquot_6.6.0 [ 1.100350] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.103054] *** VALIDATE ramfs *** [ 1.104381] *** VALIDATE hugetlbfs *** [ 1.106133] pnp: PnP ACPI init [ 1.108909] pnp: PnP ACPI: found 6 devices [ 1.125255] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.129129] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.131676] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.133961] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.136810] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.139623] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.142848] NET: Registered protocol family 2 [ 1.146337] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.151864] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.156299] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.162212] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.166113] TCP: Hash tables configured (established 65536 bind 65536) [ 1.169546] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.174409] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.178607] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.182579] NET: Registered protocol family 1 [ 1.186453] RPC: Registered named UNIX socket transport module. [ 1.188900] RPC: Registered udp transport module. [ 1.190641] RPC: Registered tcp transport module. [ 1.192672] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.195505] NET: Registered protocol family 44 [ 1.197429] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.199780] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.201966] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.204831] PCI: CLS 0 bytes, default 64 [ 1.207255] Unpacking initramfs... [ 3.211768] debug: unmapping init [mem 0xffff9fb77cc54000-0xffff9fb77ffbffff] [ 3.220432] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 3.223629] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 3.228730] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 4.044125] Initialise system trusted keyrings [ 4.046060] Key type blacklist registered [ 4.107121] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 4.117164] zbud: loaded [ 4.120144] *** VALIDATE nfs *** [ 4.121549] *** VALIDATE nfs4 *** [ 4.123245] pstore: using deflate compression [ 4.127290] Platform Keyring initialized [ 4.481470] NET: Registered protocol family 38 [ 4.483625] Key type asymmetric registered [ 4.485602] Asymmetric key parser 'x509' registered [ 4.487759] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 4.491203] io scheduler mq-deadline registered [ 4.493286] io scheduler kyber registered [ 4.495417] io scheduler bfq registered [ 4.498155] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 4.502432] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 4.505589] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 4.509178] ACPI: Power Button [PWRF] [ 4.515237] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 4.522935] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 4.536297] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 4.543782] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 4.560800] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 4.589808] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 4.620025] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 4.625389] Non-volatile memory driver v1.3 [ 4.627423] Linux agpgart interface v0.103 [ 4.660824] virtio_blk virtio1: [vda] 150400 512-byte logical blocks (77.0 MB/73.4 MiB) [ 4.663810] vda: detected capacity change from 0 to 77004800 [ 4.693422] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 4.696065] vdb: detected capacity change from 0 to 1073741824 [ 4.717922] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.720361] vdc: detected capacity change from 0 to 2621440000 [ 4.805232] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 4.808060] vdd: detected capacity change from 0 to 2621440000 [ 4.837515] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.840155] vde: detected capacity change from 0 to 4294967296 [ 4.912196] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 4.916084] vdf: detected capacity change from 0 to 4294967296 [ 4.933324] libphy: Fixed MDIO Bus: probed [ 4.942269] usbcore: registered new interface driver usbserial_generic [ 4.946934] usbserial: USB Serial support registered for generic [ 4.951031] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 4.957890] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 4.960922] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 4.965307] mousedev: PS/2 mouse device common for all mice [ 4.970022] rtc_cmos 00:05: RTC can wake from S4 [ 4.976133] rtc_cmos 00:05: registered as rtc0 [ 4.979248] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 4.982750] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 4.986522] intel_pstate: CPU model not supported [ 4.999343] hid: raw HID events driver (C) Jiri Kosina [ 5.002262] usbcore: registered new interface driver usbhid [ 5.006073] usbhid: USB HID core driver [ 5.009363] drop_monitor: Initializing network drop monitor service [ 5.011456] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 5.019574] Initializing XFRM netlink socket [ 5.020212] NET: Registered protocol family 10 [ 5.026044] Segment Routing with IPv6 [ 5.034746] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 5.037151] NET: Registered protocol family 17 [ 5.037817] mpls_gso: MPLS GSO support [ 5.066616] RAS: Correctable Errors collector initialized. [ 5.069615] AVX version of gcm_enc/dec engaged. [ 5.072169] AES CTR mode by8 optimization enabled [ 5.292765] sched_clock: Marking stable (5292526845, 0)->(6375350611, -1082823766) [ 5.299263] registered taskstats version 1 [ 5.306897] Loading compiled-in X.509 certificates [ 5.309549] zswap: loaded using pool lzo/zbud [ 5.348763] Key type big_key registered [ 5.366592] Key type encrypted registered [ 5.368873] ima: No TPM chip found, activating TPM-bypass! [ 5.370500] ima: Allocated hash algorithm: sha1 [ 5.372527] ima: No architecture policies found [ 5.373804] evm: Initialising EVM extended attributes: [ 5.375498] evm: security.selinux [ 5.376585] evm: security.ima [ 5.377505] evm: security.capability [ 5.378652] evm: HMAC attrs: 0x1 [ 5.382629] rtc_cmos 00:05: setting system clock to 2026-09-21 13:03:36 UTC (1789995816) [ 5.392137] debug: unmapping init [mem 0xffffffff94a03000-0xffffffff94bfffff] [ 5.395851] debug: unmapping init [mem 0xffffffff93782000-0xffffffff93a58fff] [ 5.407152] Write protecting the kernel read-only data: 28672k [ 5.413107] debug: unmapping init [mem 0xffffffff91e03000-0xffffffff91ffffff] [ 5.419882] debug: unmapping init [mem 0xffffffff92714000-0xffffffff927fffff] [ 5.480466] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 5.493653] systemd[1]: Detected virtualization kvm. [ 5.496172] systemd[1]: Detected architecture x86-64. [ 5.498847] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 5.530247] systemd[1]: No hostname configured. [ 5.532183] systemd[1]: Set hostname to . [ 5.534301] random: systemd: uninitialized urandom read (16 bytes read) [ 5.536641] systemd[1]: Initializing machine ID from random generator. [ 5.789587] random: systemd: uninitialized urandom read (16 bytes read) [ 5.791933] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 5.803457] random: systemd: uninitialized urandom read (16 bytes read) [ 5.806248] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 5.813618] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 7.386322] device-mapper: uevent: version 1.0.3 [ 7.388391] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted Kernel Configuration File System. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 9.572653] scsi host0: ata_piix [ 9.652962] random: fast init done [ 9.681187] virtio_net virtio0 ens2: renamed from eth0 [ 9.761869] scsi host1: ata_piix [ 9.771026] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.782784] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 14.959334] random: crng init done [ 14.960698] random: 7 urandom warning(s) missed due to ratelimiting [ 17.853734] dracut-initqueue[589]: RTNETLINK answers: File exists Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ 19.159701] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Reached target Remote File Systems. [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 21.509803] printk: systemd: 26 output lines suppressed due to ratelimiting [ 22.054811] SELinux: Disabled at runtime. [ 22.153860] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 22.169966] systemd[1]: Detected virtualization kvm. [ 22.172749] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.309166] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.313291] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.323299] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.327979] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.331873] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.340511] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.349487] systemd[1]: Created slice system-getty.slice. [ OK ] Created slice system-getty.slice. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. Mounting Huge Pages File System... [ OK ] Listening on udev Control Socket. [ 23.613190] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. Mounting POSIX Message Queue File System... [ OK ] Reached target Slices. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. Starting Remount Root and Kernel File Systems... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started udev Coldplug all Devices. [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 24.771783] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 25.933813] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 25.974979] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.581219] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 27.054778] EDAC sbridge: Ver: 1.1.2 [ 30.794533] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.260193] NFS: Registering the id_resolver key type [ 31.262702] Key type id_resolver registered [ 31.264609] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started irqbalance daemon. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started D-Bus System Message Bus. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg344-server login: [ 78.611292] libcfs: loading out-of-tree module taints kernel. [ 78.643798] Key type ._llcrypt registered [ 78.646321] Key type .llcrypt registered [ 78.718626] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 89.015311] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 89.693948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 89.702583] alg: No test for adler32 (adler32-zlib) [ 90.806240] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 91.190798] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 92.840239] Key type lgssc registered [ 93.557789] Lustre: Echo OBD driver; http://www.lustre.org/ [ 102.276878] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 122.856617] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 129.913165] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 129.929648] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 131.059079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 131.073208] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 131.123122] Lustre: lustre-MDT0000: new disk, initializing [ 131.167407] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 131.178291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 133.318724] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 144.333458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 144.466753] Lustre: 6480:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 144.572847] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 144.578717] Lustre: Skipped 1 previous similar message [ 144.644756] Lustre: lustre-MDT0001: new disk, initializing [ 144.708592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 144.754751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 144.772166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 149.904870] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 157.109173] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 167.626170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 167.857840] Lustre: lustre-OST0000: new disk, initializing [ 167.865429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 167.872508] Lustre: 8416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 167.940090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 175.179909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 175.190917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 175.276284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 175.484689] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.188447] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 196.479485] Lustre: lustre-OST0001: new disk, initializing [ 196.486380] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 196.491465] Lustre: 9488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 196.559641] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 201.025034] hrtimer: interrupt took 3027093 ns [ 206.390508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 206.402512] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 206.479851] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 207.027643] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.004700] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 232.676637] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 239.215038] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing check_logdir /tmp/testlogs/ [ 244.516695] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing yml_node [ 249.230351] Lustre: DEBUG MARKER: Client: 2.17.58.63 [ 251.946158] Lustre: DEBUG MARKER: MDS: 2.17.58.63 [ 254.668686] Lustre: DEBUG MARKER: OSS: 2.17.58.63 [ 256.204800] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 21 09:07:46 EDT 2026 [ 272.538580] Lustre: DEBUG MARKER: only running tests: 121-999 [ 273.862467] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 275.227124] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 293.349177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 293.353665] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 293.372120] Lustre: Skipped 2 previous similar messages [ 293.376328] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 293.376570] Lustre: Skipped 3 previous similar messages [ 296.166591] Lustre: server umount lustre-MDT0000 complete [ 299.592952] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996110 with bad export cookie 6043518607751072217 [ 299.598844] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 299.605340] LustreError: 6473:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 303.595601] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 303.598164] LustreError: 6488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 303.598905] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 303.611245] Lustre: Skipped 2 previous similar messages [ 303.647518] LustreError: 6488:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 304.614378] LustreError: 6487:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 304.614997] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 304.648109] LustreError: 6487:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 304.680243] Lustre: Skipped 2 previous similar messages [ 306.026851] Lustre: server umount lustre-MDT0001 complete [ 310.200567] Lustre: server umount lustre-OST0000 complete [ 313.897866] Lustre: server umount lustre-OST0001 complete [ 321.277852] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 331.600359] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 371.316835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 371.539849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 371.597852] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 371.720442] Lustre: lustre-MDT0000: new disk, initializing [ 371.826253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 371.851800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.744124] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 388.110154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.273824] Lustre: 15095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 388.341145] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 388.345658] Lustre: Skipped 1 previous similar message [ 388.413737] Lustre: lustre-MDT0001: new disk, initializing [ 388.540976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 388.590433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 388.601695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 393.260932] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 402.443483] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 408.079587] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 414.588858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 414.902632] Lustre: lustre-OST0000: new disk, initializing [ 414.907820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 414.917511] Lustre: 17064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 415.056085] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 416.726234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 416.735125] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 416.870405] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 421.393232] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 431.374461] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 436.178786] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 436.393294] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 441.158740] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 441.426122] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 447.460405] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 447.470229] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 447.470339] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 447.480628] Lustre: Skipped 1 previous similar message [ 447.501621] Lustre: Skipped 1 previous similar message [ 449.499902] Lustre: server umount lustre-OST0000 complete [ 455.144441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 455.163224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 455.181430] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 459.813657] Lustre: server umount lustre-MDT0000 complete [ 462.818015] LustreError: 15106:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 462.846973] LustreError: 15106:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 463.543612] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996274 with bad export cookie 6043518607751073750 [ 463.554343] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 463.558987] LustreError: 15087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 464.078220] Lustre: server umount lustre-MDT0001 complete [ 471.263535] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 09:11:20 (1789996280) [ 491.379221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 491.864449] LustreError: 19941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 491.930252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 496.257246] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 497.126331] LustreError: 19941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 504.920837] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 509.679399] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 517.179142] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 523.366139] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 525.685903] Lustre: Failing over lustre-MDT0000 [ 525.801506] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 525.810570] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.818880] Lustre: Skipped 1 previous similar message [ 525.825978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 525.829779] Lustre: Skipped 2 previous similar messages [ 526.010930] Lustre: server umount lustre-MDT0000 complete [ 527.851892] LustreError: 19941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 527.871116] LustreError: 19941:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 543.203911] Lustre: 3621:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789996338/real 1789996338] req@ffff9fb7fc1c8700 x1876946743098624/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789996354 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 543.220723] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 547.591589] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 548.328599] LustreError: 19935:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 548.363697] LustreError: 19935:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 553.445759] LustreError: 3617:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9fb80187e680 x1876946743101440/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 559.589512] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 559.596778] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 559.651890] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 560.525667] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 572.147078] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 580.066527] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 580.067861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 580.071358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 580.083412] Lustre: Skipped 2 previous similar messages [ 582.452727] Lustre: server umount lustre-MDT0000 complete [ 585.186925] LustreError: 20668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.199181] LustreError: 20668:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 585.706587] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996396 with bad export cookie 6043518607751075164 [ 585.708241] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 585.713577] LustreError: 19921:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 585.947802] Lustre: server umount lustre-MDT0001 complete [ 593.618883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 594.089784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 594.096621] Lustre: Skipped 2 previous similar messages [ 598.537959] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 606.519102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 611.283357] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 618.569552] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 624.557266] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 627.030960] Lustre: Failing over lustre-MDT0000 [ 627.051099] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 627.074962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 627.078173] Lustre: Skipped 2 previous similar messages [ 627.413070] Lustre: server umount lustre-MDT0000 complete [ 647.650067] Lustre: 3619:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789996442/real 1789996442] req@ffff9fb800f81f80 x1876946743129728/t0(0) o400->MGC192.168.203.144@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789996458 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 647.677491] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 648.067663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.809967] LustreError: 24006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 651.833701] LustreError: 24006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 657.323203] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 662.501207] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 662.516193] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 662.517831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 662.517838] Lustre: Skipped 1 previous similar message [ 662.550920] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 672.782983] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 677.140177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.162827] Lustre: Skipped 2 previous similar messages [ 683.567455] Lustre: server umount lustre-MDT0000 complete [ 687.321577] LustreError: 23257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996498 with bad export cookie 6043518607751076102 [ 687.336148] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 687.675077] Lustre: server umount lustre-MDT0001 complete [ 697.504202] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 09:15:07 (1789996507) [ 715.453569] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 726.058727] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 769.897479] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 782.160621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 782.393679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 782.420598] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 782.525406] Lustre: lustre-MDT0000: new disk, initializing [ 782.625836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 782.631500] Lustre: Skipped 2 previous similar messages [ 782.653604] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 787.740225] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 798.604229] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 798.725457] Lustre: 29653:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 798.956917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 804.506460] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 814.314397] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 820.921474] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 827.865705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 828.153786] Lustre: 31624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 829.382290] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 829.449663] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 829.524453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 829.546887] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 835.248781] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 844.939415] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 850.824450] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 875.493775] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 875.501788] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 875.510306] LustreError: Skipped 1 previous similar message [ 875.511976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 875.520969] Lustre: Skipped 1 previous similar message [ 875.544120] Lustre: Skipped 5 previous similar messages [ 887.266187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 887.474178] Lustre: server umount lustre-OST0000 complete [ 897.888777] Lustre: server umount lustre-MDT0000 complete [ 901.091392] LustreError: 30464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 901.111479] LustreError: 30464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 902.065514] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996713 with bad export cookie 6043518607751076508 [ 902.079663] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.087080] LustreError: 29646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 902.573221] Lustre: server umount lustre-MDT0001 complete [ 915.824525] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 919.475708] Key type lgssc unregistered [ 919.759845] LNet: 33604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 919.768702] LNetError: 33604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 919.799935] LNet: Removed LNI 192.168.203.144@tcp [ 920.972214] Key type .llcrypt unregistered [ 920.974867] Key type ._llcrypt unregistered [ 950.904183] Key type ._llcrypt registered [ 950.909816] Key type .llcrypt registered [ 951.094876] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 952.355773] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 952.495752] alg: No test for adler32 (adler32-zlib) [ 953.562847] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 953.814870] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 955.544368] Key type lgssc registered [ 956.704869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 965.234332] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 965.275350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 966.879163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.085279] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 976.361522] LustreError: 35034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.387758] LustreError: 35034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 981.044250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 981.507066] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 986.689619] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 994.683035] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1000.686162] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1007.273038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.702333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1011.858947] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1013.119837] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1019.558413] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1026.165753] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1037.232944] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 09:20:46 (1789996846) [ 1039.370041] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1041.032183] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1050.736702] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 09:20:59 (1789996859) [ 1065.297733] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 09:21:14 (1789996874) [ 1066.087237] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1075.229758] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1075.235464] Lustre: Skipped 2 previous similar messages [ 1077.033558] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 09:21:26 (1789996886) [ 1080.713168] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1080.723232] Lustre: Skipped 3 previous similar messages [ 1102.075355] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1102.083378] Lustre: Skipped 6 previous similar messages [ 1105.322610] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 09:21:54 (1789996914) [ 1108.374969] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1111.575494] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1120.339791] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 09:22:10 (1789996930) [ 1122.319680] Lustre: Modifying parameter general.jobid_name=TESTNAME1 in log params [ 1122.332601] Lustre: Skipped 3 previous similar messages [ 1135.357893] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 09:22:25 (1789996945) [ 1150.434084] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1150.448634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1150.473216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.531982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1154.544402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1154.547225] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1154.576164] Lustre: Skipped 1 previous similar message [ 1159.650429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1164.256167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1164.524358] Lustre: server umount lustre-OST0000 complete [ 1169.910355] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1169.932146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1169.936451] Lustre: Skipped 2 previous similar messages [ 1175.016700] LustreError: 35034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1175.062744] Lustre: server umount lustre-MDT0000 complete [ 1178.539381] LustreError: 35015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789996989 with bad export cookie 2303500334144441724 [ 1178.546163] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1178.550402] LustreError: 35015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1178.823691] Lustre: server umount lustre-MDT0001 complete [ 1188.822177] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1191.386928] Key type lgssc unregistered [ 1191.633421] LNet: 42893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.642715] LNetError: 42893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.671193] LNet: Removed LNI 192.168.203.144@tcp [ 1192.437460] Key type .llcrypt unregistered [ 1192.441563] Key type ._llcrypt unregistered [ 1209.554883] Key type ._llcrypt registered [ 1209.557318] Key type .llcrypt registered [ 1209.639942] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 1225.136341] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1226.110898] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1226.121579] alg: No test for adler32 (adler32-zlib) [ 1227.306587] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1227.578457] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1229.280686] Key type lgssc registered [ 1230.621432] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1264.202076] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1264.223875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1265.511169] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1265.546759] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1265.669871] Lustre: lustre-MDT0000: new disk, initializing [ 1265.789745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1265.820782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1270.197701] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1279.792798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.900240] Lustre: 46061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1279.953144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1279.959318] Lustre: Skipped 1 previous similar message [ 1280.041315] Lustre: lustre-MDT0001: new disk, initializing [ 1280.160307] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1280.184662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1280.191137] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1283.796323] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1292.651090] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1297.872324] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1303.456900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1303.659086] Lustre: lustre-OST0000: new disk, initializing [ 1303.663274] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1303.672435] Lustre: 48028:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1303.744155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1308.943502] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1311.272427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1311.290338] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1311.383598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1316.600980] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1320.074619] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1320.224736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1323.684400] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1323.869770] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1326.563493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1326.568243] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1326.576316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1327.083243] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1327.083947] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1327.107352] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1331.697202] Lustre: server umount lustre-OST0000 complete [ 1336.801818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1336.817235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1336.842215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1336.845257] Lustre: Skipped 1 previous similar message [ 1340.887655] Lustre: server umount lustre-MDT0000 complete [ 1342.433641] LustreError: 46067:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1342.468617] LustreError: 46067:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1344.546157] LustreError: 46053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997155 with bad export cookie 16761331981913699746 [ 1344.552681] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1344.567403] LustreError: 46053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1344.863368] Lustre: server umount lustre-MDT0001 complete [ 1354.221827] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 09:26:03 (1789997163) [ 1360.388731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1360.913755] LustreError: 50488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1360.999706] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1365.348130] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1366.504984] LustreError: 50489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.624314] LustreError: 50488:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1372.834580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1373.128578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1376.651562] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1382.882439] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1387.037927] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1392.130486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1392.408545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1397.695862] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1404.289872] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1412.489607] LustreError: 53019:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1413.272354] Lustre: Modifying parameter general.timeout=1 in log params [ 1414.947617] Lustre: Modifying parameter general.timeout=3 in log params [ 1414.950398] Lustre: Skipped 1 previous similar message [ 1417.240613] Lustre: Modifying parameter general.timeout=6 in log params [ 1417.243763] Lustre: Skipped 2 previous similar messages [ 1421.933954] Lustre: Modifying parameter general.timeout=12 in log params [ 1421.942669] Lustre: Skipped 5 previous similar messages [ 1430.102596] Lustre: Modifying parameter general.timeout=22 in log params [ 1430.107385] Lustre: Skipped 9 previous similar messages [ 1446.600053] Lustre: Modifying parameter general.timeout=46 in log params [ 1446.604328] Lustre: Skipped 23 previous similar messages [ 1478.668088] Lustre: Modifying parameter general.timeout=88 in log params [ 1478.671211] Lustre: Skipped 41 previous similar messages [ 1517.921666] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 09:28:47 (1789997327) [ 1520.327816] Lustre: 59947:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1522.088929] Lustre: 60045:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1525.256849] Lustre: 60257:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1525.264405] Lustre: 60257:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1542.916333] Lustre: Disabling parameter general.ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 in log params [ 1542.922285] Lustre: Skipped 60 previous similar messages [ 1549.257310] Lustre: 61820:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1549.267482] Lustre: 61820:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1549.286738] LustreError: 61820:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1552.442623] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 09:29:22 (1789997362) [ 1565.161491] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.163684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1565.176395] Lustre: Skipped 3 previous similar messages [ 1565.182813] Lustre: Skipped 3 previous similar messages [ 1569.159393] Lustre: server umount lustre-MDT0000 complete [ 1570.274808] LustreError: 50485:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1570.289254] LustreError: 50485:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1572.375076] LustreError: 50469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997383 with bad export cookie 16761331981913700621 [ 1572.382874] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1572.388494] LustreError: 50469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1572.629624] Lustre: server umount lustre-MDT0001 complete [ 1586.051811] Lustre: server umount lustre-OST0000 complete [ 1609.274262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1609.506827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1609.770967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.001306] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1620.304070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1620.397398] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1620.423345] Lustre: 64350:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1624.656610] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1631.520629] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1636.464379] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1642.323324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1642.474193] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1642.707796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1642.711757] Lustre: Skipped 1 previous similar message [ 1647.736728] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1654.417993] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1655.977351] Lustre: 66919:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1662.439110] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1662.439979] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1662.451805] LustreError: Skipped 1 previous similar message [ 1662.452487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1662.452491] Lustre: Skipped 1 previous similar message [ 1662.488885] Lustre: Skipped 1 previous similar message [ 1667.559987] LustreError: 66200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1667.572682] LustreError: 66200:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1667.648196] Lustre: server umount lustre-OST0000 complete [ 1672.161342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1672.170661] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1672.185681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1672.191339] Lustre: Skipped 1 previous similar message [ 1677.062550] Lustre: server umount lustre-MDT0000 complete [ 1680.314543] LustreError: 64343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997491 with bad export cookie 16761331981913703519 [ 1680.327792] LustreError: 64343:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1680.330754] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1680.698588] Lustre: server umount lustre-MDT0001 complete [ 1690.820484] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1693.022945] Key type lgssc unregistered [ 1693.280658] LNet: 68123:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1693.292668] LNetError: 68123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1694.317633] LNet: Removed LNI 192.168.203.144@tcp [ 1694.933958] Key type .llcrypt unregistered [ 1694.936056] Key type ._llcrypt unregistered [ 1715.382866] Key type ._llcrypt registered [ 1715.386372] Key type .llcrypt registered [ 1715.475308] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1716.403272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1716.501198] alg: No test for adler32 (adler32-zlib) [ 1717.452734] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1717.682826] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1719.401568] Key type lgssc registered [ 1720.532719] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1726.090107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1726.110619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.424761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.706811] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1737.231390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1737.631418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1740.711287] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1746.184549] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1750.460585] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1755.375992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1755.752889] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1759.930690] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1764.839093] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1781.478169] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1788.900779] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.904353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1788.904815] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1788.916534] Lustre: Skipped 2 previous similar messages [ 1788.919921] Lustre: Skipped 1 previous similar message [ 1794.020941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1794.023822] Lustre: Skipped 1 previous similar message [ 1794.331148] Lustre: server umount lustre-MDT0000 complete [ 1797.442611] LustreError: 69529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997608 with bad export cookie 309747578614694462 [ 1797.446256] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1797.449140] LustreError: 69529:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1797.759762] Lustre: server umount lustre-MDT0001 complete [ 1810.140155] Lustre: server umount lustre-OST0000 complete [ 1829.708766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1829.904268] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1829.927763] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1830.135831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1833.248676] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1839.128357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1839.208632] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1839.231948] Lustre: 74440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1839.592940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1842.187906] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1847.094772] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1851.345235] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1855.489650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1855.610965] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1855.869937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1859.645805] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1868.124699] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1869.932656] Lustre: 77013:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1872.931909] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1879.528738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1879.543565] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.555467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1884.129431] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1884.149268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.177992] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1886.066672] Lustre: server umount lustre-OST0000 complete [ 1894.882878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1894.893175] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1894.895341] Lustre: Skipped 1 previous similar message [ 1894.916734] Lustre: Skipped 2 previous similar messages [ 1899.209471] Lustre: server umount lustre-MDT0000 complete [ 1900.004221] LustreError: 74446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1900.015990] LustreError: 74446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1902.548123] LustreError: 74430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997713 with bad export cookie 309747578614695295 [ 1902.554636] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1902.557607] LustreError: 74430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1902.884525] Lustre: server umount lustre-MDT0001 complete [ 1911.838713] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 1914.202335] Key type lgssc unregistered [ 1914.434182] LNet: 78224:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1914.439129] LNetError: 78224:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1914.455744] LNet: Removed LNI 192.168.203.144@tcp [ 1915.127270] Key type .llcrypt unregistered [ 1915.129376] Key type ._llcrypt unregistered [ 1938.997246] Key type ._llcrypt registered [ 1939.000700] Key type .llcrypt registered [ 1939.110334] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 1940.057920] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1940.157835] alg: No test for adler32 (adler32-zlib) [ 1941.207161] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 1941.432281] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 1943.072219] Key type lgssc registered [ 1944.141767] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1951.867393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1951.913698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1953.385469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1957.632464] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1962.984812] LustreError: 79653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1963.001642] LustreError: 79653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1964.741587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.192640] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1968.721446] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1975.227567] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1980.153963] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1985.398881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1985.888833] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1991.038741] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 1997.839561] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2003.858613] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2004.551784] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2006.047238] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2006.054151] Lustre: Skipped 1 previous similar message [ 2008.138691] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2008.142121] Lustre: Skipped 2 previous similar messages [ 2012.559024] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2012.563529] Lustre: Skipped 5 previous similar messages [ 2020.800401] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2020.805234] Lustre: Skipped 10 previous similar messages [ 2037.381915] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 2037.386539] Lustre: Skipped 22 previous similar messages [ 2069.665983] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 2069.670797] Lustre: Skipped 42 previous similar messages [ 2083.186471] Lustre: 87741:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 2085.357125] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 09:38:15 (1789997895) [ 2086.691943] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 2088.235829] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 09:38:18 (1789997898) [ 2092.513173] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2092.524121] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2092.536128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.586069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2095.597748] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2095.606506] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.458877] Lustre: server umount lustre-OST0000 complete [ 2100.708893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2100.728110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2100.733719] Lustre: Skipped 2 previous similar messages [ 2105.827335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2105.838642] Lustre: Skipped 1 previous similar message [ 2106.639685] Lustre: server umount lustre-MDT0000 complete [ 2109.410909] LustreError: 83621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789997920 with bad export cookie 4584743430502538535 [ 2109.412775] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.420112] LustreError: 83621:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2109.619623] Lustre: server umount lustre-MDT0001 complete [ 2117.810535] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2119.889561] Key type lgssc unregistered [ 2120.106317] LNet: 88989:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2120.112318] LNetError: 88989:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2120.129264] LNet: Removed LNI 192.168.203.144@tcp [ 2120.729137] Key type .llcrypt unregistered [ 2120.736876] Key type ._llcrypt unregistered [ 2123.859684] Key type ._llcrypt registered [ 2123.861733] Key type .llcrypt registered [ 2123.922901] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 2126.681651] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules [ 2127.367670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2127.387771] alg: No test for adler32 (adler32-zlib) [ 2128.335329] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 2128.553513] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2130.201193] Key type lgssc registered [ 2131.015889] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2131.980946] LustreError: 90200:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2148.118873] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2154.300610] LustreError: 90200:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2154.965404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2154.976964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2156.302142] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2160.007021] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2166.758986] LustreError: 91230:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.768073] LustreError: 91230:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2166.918737] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 09:39:37 (1789997977) [ 2170.154144] LustreError: 91262:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2170.165212] LustreError: 91262:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2170.171191] LustreError: 91262:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 0, failed: rc = -5 [ 2172.400814] Lustre: server umount lustre-MDT0000 complete [ 2180.497632] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 2182.129839] Key type lgssc unregistered [ 2182.306209] LNet: 92754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2182.312915] LNetError: 92754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2182.327392] LNet: Removed LNI 192.168.203.144@tcp [ 2182.787333] Key type .llcrypt unregistered [ 2182.789675] Key type ._llcrypt unregistered [ 2195.220426] Key type ._llcrypt registered [ 2195.222805] Key type .llcrypt registered [ 2195.296595] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2195.809244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2195.828206] alg: No test for adler32 (adler32-zlib) [ 2196.771585] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 2196.932606] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 2198.544200] Key type lgssc registered [ 2199.361616] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2204.363316] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2204.379793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2205.673666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2207.915688] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2212.658955] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2213.022130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2215.482774] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2219.888716] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2223.436714] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2227.618793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2228.042268] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2231.631583] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2235.934608] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2268.016636] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 09:41:18 (1789998078) [ 2269.201079] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2270.431710] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 09:41:20 (1789998080) [ 2273.250906] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2273.268119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2273.274639] Lustre: Skipped 2 previous similar messages [ 2278.371824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2278.376176] Lustre: Skipped 2 previous similar messages [ 2279.159963] Lustre: server umount lustre-MDT0000 complete [ 2281.438884] LustreError: 93945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998092 with bad export cookie 7260277639922101642 [ 2281.442248] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.445889] LustreError: 93945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2281.622474] Lustre: server umount lustre-MDT0001 complete [ 2293.098896] Lustre: server umount lustre-OST0000 complete [ 2300.107267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2300.355736] LustreError: 98231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2300.372376] LustreError: 98231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2300.475189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.233640] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2305.505547] LustreError: 98232:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.409896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2308.821556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2311.077582] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2315.140912] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2318.489338] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2326.646028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.766613] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2326.772872] LustreError: 98221:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2326.789730] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2326.795637] LustreError: 100311:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2326.800504] LustreError: 100311:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2326.804249] LustreError: 100311:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2326.845906] Lustre: server umount lustre-OST0000 complete [ 2326.852658] LustreError: 100311:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2332.414800] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2332.547591] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2332.554597] LustreError: 98221:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2332.569338] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2332.576872] LustreError: 100772:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2332.584506] LustreError: 100772:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2332.589713] LustreError: 100772:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2332.627419] Lustre: server umount lustre-OST0000 complete [ 2332.632977] LustreError: 100772:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2338.085828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2338.188489] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2338.193282] Lustre: Found index 0 for lustre-OST0000, updating log [ 2338.197330] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2338.243655] Lustre: lustre-OST0000: new disk, initializing [ 2338.246661] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2338.251256] Lustre: 101308:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2338.440352] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2342.160923] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2344.243369] Lustre: Failing over lustre-OST0000 [ 2344.331660] Lustre: server umount lustre-OST0000 complete [ 2349.538703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2349.542347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.548321] Lustre: Skipped 2 previous similar messages [ 2349.550456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.585457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2351.586260] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.697060] Lustre: server umount lustre-MDT0000 complete [ 2353.641156] LustreError: 98231:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2353.655925] LustreError: 98231:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2354.940908] LustreError: 98211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998166 with bad export cookie 7260277639922104372 [ 2354.943545] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2354.949146] LustreError: 98211:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2355.114668] Lustre: server umount lustre-MDT0001 complete [ 2360.543122] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 09:42:50 (1789998170) [ 2371.788125] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2377.470972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.713284] LustreError: 103649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2377.776598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2380.256076] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2382.818871] LustreError: 103650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2385.351800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.218819] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2389.946872] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2393.917234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2397.841124] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2402.777971] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2402.823679] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2402.865727] Lustre: lustre-OST0001: new disk, initializing [ 2402.868878] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2402.872735] Lustre: 105965:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2406.104451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2406.112688] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2406.172410] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2406.340648] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2411.340601] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2415.589537] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2421.730425] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2421.740240] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2421.745309] Lustre: Skipped 1 previous similar message [ 2421.757156] Lustre: Skipped 3 previous similar messages [ 2427.483350] Lustre: server umount lustre-MDT0001 complete [ 2428.529526] LustreError: 106443:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2438.844522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2438.926682] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2438.935236] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2438.998117] Lustre: 103640:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2439.801756] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2439.806892] Lustre: Skipped 3 previous similar messages [ 2440.323193] LustreError: 107504:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2440.332926] LustreError: 107504:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2440.339561] LustreError: 103642:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2440.460882] LustreError: 107513:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -17 [ 2440.472276] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2440.472276] [ 2445.307253] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2445.776804] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2461.922386] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 09:44:30 (1789998270) [ 2493.609092] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2510.578211] Lustre: 109603:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2530.936358] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2533.757628] Lustre: 110737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2618.849728] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2618.857379] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2618.876813] Lustre: Skipped 1 previous similar message [ 2618.882235] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2618.896913] Lustre: Skipped 4 previous similar messages [ 2622.996669] Lustre: server umount lustre-MDT0000 complete [ 2624.481257] LustreError: 103646:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2624.503345] LustreError: 103646:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 2625.676626] LustreError: 106683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998436 with bad export cookie 7260277639922105289 [ 2625.678531] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2625.689455] LustreError: 106683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2625.899294] Lustre: server umount lustre-MDT0001 complete [ 2639.953403] Lustre: server umount lustre-OST0000 complete [ 2653.277463] Lustre: server umount lustre-OST0001 complete [ 2658.073775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2666.380902] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2672.862467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2680.623154] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2696.658074] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2704.394614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2704.410851] Lustre: lustre-MDT0000: reset Object Index mappings [ 2704.822335] LustreError: 114689:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2704.938607] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2708.337478] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2714.637728] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2714.668417] Lustre: lustre-MDT0001: reset Object Index mappings [ 2718.257299] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2720.603507] Lustre: 115835:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2725.716614] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2729.397345] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:195 to 0x280000400:225) [ 2729.399620] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2730.863156] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2736.784707] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2741.909816] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2742.273193] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:161 to 0x2c0000401:193) [ 2742.296883] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2749.132562] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2752.961727] Lustre: 117714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2801.146988] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 09:50:11 (1789998611) [ 2827.747134] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2827.755175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2827.768550] Lustre: Skipped 3 previous similar messages [ 2827.772045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2827.780295] Lustre: Skipped 4 previous similar messages [ 2833.131992] Lustre: server umount lustre-MDT0000 complete [ 2834.407276] LustreError: 117734:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2834.421726] LustreError: 117734:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 2836.246964] LustreError: 114669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789998647 with bad export cookie 7260277639922152210 [ 2836.259241] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2842.799712] Lustre: server umount lustre-MDT0001 complete [ 2852.266624] Lustre: server umount lustre-OST0000 complete [ 2861.901109] Lustre: server umount lustre-OST0001 complete [ 2868.222525] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 2877.472673] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2909.378374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2909.656590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2909.700023] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2909.794865] Lustre: lustre-MDT0000: new disk, initializing [ 2909.848745] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2909.866323] Lustre: Skipped 3 previous similar messages [ 2909.894770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2913.558902] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2919.089279] Lustre: server umount lustre-MDT0000 complete [ 2926.720239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2927.016583] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2927.032570] Lustre: 121979:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2927.046776] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2930.905858] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2934.975922] Lustre: server umount lustre-MDT0000 complete [ 2942.048278] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 09:52:32 (1789998752) [ 2943.699925] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2945.243129] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 09:52:35 (1789998755) [ 2960.707878] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 2968.328085] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2972.466109] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2979.623112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2979.735179] Lustre: 124029:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2979.794439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2979.799384] Lustre: Skipped 1 previous similar message [ 2979.863848] Lustre: lustre-MDT0001: new disk, initializing [ 2979.961266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2979.970977] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2983.288882] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 2987.287756] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2993.439506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2993.762760] Lustre: lustre-OST0000: new disk, initializing [ 2993.772877] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2993.782748] Lustre: 125606:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2995.147441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2995.164190] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2995.300729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2999.700877] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3009.238058] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3009.419987] Lustre: lustre-OST0001: new disk, initializing [ 3009.423214] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3009.428542] Lustre: 126475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3010.738208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3010.749598] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3010.848537] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3014.706684] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3023.103478] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 3026.423709] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3400.888695] Lustre: 132893:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3400.894896] Lustre: 132893:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3400.899201] Lustre: 132893:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3400.903317] Lustre: 132893:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3400.907462] Lustre: 132893:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3400.912067] Lustre: 132893:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3703.243682] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 10:05:12 (1789999512) [ 3718.626705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3718.639970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.647215] Lustre: Skipped 8 previous similar messages [ 3718.658568] Lustre: Skipped 11 previous similar messages [ 3723.747266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3723.759484] Lustre: Skipped 2 previous similar messages [ 3724.570146] Lustre: server umount lustre-MDT0000 complete [ 3727.972631] LustreError: 132906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789999539 with bad export cookie 7260277639922220250 [ 3727.984800] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3727.998900] LustreError: 132906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3728.864935] LustreError: 124036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3728.877905] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3728.900275] LustreError: 124036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3728.921926] Lustre: Skipped 1 previous similar message [ 3732.899160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3732.912928] Lustre: Skipped 3 previous similar messages [ 3734.446748] Lustre: server umount lustre-MDT0001 complete [ 3744.222622] Lustre: server umount lustre-OST0000 complete [ 3754.366799] Lustre: server umount lustre-OST0001 complete [ 3760.471200] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 3769.135789] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 3809.646027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3809.907941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3809.932576] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3810.035182] Lustre: lustre-MDT0000: new disk, initializing [ 3810.194357] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3810.210734] Lustre: Skipped 5 previous similar messages [ 3810.241540] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3814.758399] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3824.561209] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3824.727607] Lustre: 135959:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3824.782638] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3824.786716] Lustre: Skipped 1 previous similar message [ 3824.894898] Lustre: lustre-MDT0001: new disk, initializing [ 3825.103179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3825.115650] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3829.303181] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3838.304923] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3844.067713] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3850.192354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3850.415749] Lustre: lustre-OST0000: new disk, initializing [ 3850.419961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3850.425668] Lustre: 137930:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3850.507691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3850.514578] Lustre: Skipped 1 previous similar message [ 3852.127291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3852.133575] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3852.237627] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3856.133487] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 3867.397368] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3878.670260] Lustre: lustre-MDD0000: changelog on [ 3881.125862] Lustre: lustre-MDD0001: changelog on [ 5903.372082] Lustre: lustre-MDD0001: changelog off [ 5905.636649] Lustre: 140086:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5906.534137] Lustre: lustre-MDD0000: changelog off [ 5915.624437] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5915.629058] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5915.638810] LustreError: Skipped 1 previous similar message [ 5915.643186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5915.654134] Lustre: Skipped 1 previous similar message [ 5920.195143] Lustre: server umount lustre-OST0000 complete [ 5925.858610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5925.873597] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5925.875470] Lustre: Skipped 1 previous similar message [ 5925.890641] Lustre: Skipped 2 previous similar messages [ 5930.981535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5932.076977] Lustre: server umount lustre-MDT0000 complete [ 5935.075130] LustreError: 135967:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5935.083157] LustreError: 135967:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 5937.029853] LustreError: 136767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790001748 with bad export cookie 7260277639923033797 [ 5937.036979] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5937.055930] LustreError: 136767:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5937.467302] Lustre: server umount lustre-MDT0001 complete [ 5948.952573] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 5952.430687] Key type lgssc unregistered [ 5952.789667] LNet: 147077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5952.806911] LNetError: 147077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5952.837359] LNet: Removed LNI 192.168.203.144@tcp [ 5953.996282] Key type .llcrypt unregistered [ 5954.000431] Key type ._llcrypt unregistered [ 5977.838556] Key type ._llcrypt registered [ 5977.845665] Key type .llcrypt registered [ 5978.025761] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 5996.114026] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 5997.683272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5997.713703] alg: No test for adler32 (adler32-zlib) [ 5998.860364] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 5999.215432] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 6000.944609] Key type lgssc registered [ 6002.099735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6038.989415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6039.013807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6040.303359] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6040.357857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6040.444372] Lustre: lustre-MDT0000: new disk, initializing [ 6040.527266] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6040.550583] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6044.651052] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6055.082463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6055.195866] Lustre: 150296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6055.245796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6055.248168] Lustre: Skipped 1 previous similar message [ 6055.310493] Lustre: lustre-MDT0001: new disk, initializing [ 6055.419705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6055.478262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6055.490856] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6060.090773] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6069.630119] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6074.997936] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6081.846864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6082.077170] Lustre: lustre-OST0000: new disk, initializing [ 6082.082666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6082.088762] Lustre: 152263:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6082.183658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6087.992380] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6091.322494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6091.327993] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6091.429138] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6096.844897] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6103.469799] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_module obdecho/obdecho [ 6105.246556] LustreError: 153265:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 6105.254851] LustreError: 153265:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 6107.016482] LustreError: 153362:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 6121.953977] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6121.970545] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6121.998280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6124.861614] Lustre: server umount lustre-MDT0000 complete [ 6128.490607] LustreError: 150287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790001939 with bad export cookie 12323032830144859748 [ 6128.502670] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6128.504023] LustreError: 150287:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6128.843285] Lustre: server umount lustre-MDT0001 complete [ 6143.418211] Lustre: server umount lustre-OST0000 complete [ 6164.972299] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6175.866491] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6209.879823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6210.138137] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6210.174488] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6210.244491] Lustre: lustre-MDT0000: new disk, initializing [ 6210.315707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6210.333762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6214.410001] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6223.636398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6223.721325] Lustre: 157079:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6223.792865] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6223.799264] Lustre: Skipped 1 previous similar message [ 6223.900765] Lustre: lustre-MDT0001: new disk, initializing [ 6224.029905] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6224.083081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6224.105366] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6228.219821] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6238.136627] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6243.428680] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6250.031520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6250.300988] Lustre: lustre-OST0000: new disk, initializing [ 6250.315619] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6250.323505] Lustre: 159045:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6250.396904] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6256.642939] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6258.757966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6258.771403] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6258.826960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6265.907347] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6270.618618] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6270.822727] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6275.726466] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6275.961511] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6279.652620] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6279.655508] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6279.663183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6279.663193] Lustre: Skipped 3 previous similar messages [ 6279.669032] LustreError: Skipped 1 previous similar message [ 6279.708240] Lustre: Skipped 3 previous similar messages [ 6283.990552] Lustre: server umount lustre-OST0000 complete [ 6289.889729] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6289.896955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6289.915683] Lustre: Skipped 1 previous similar message [ 6289.916170] Lustre: Skipped 1 previous similar message [ 6293.280474] Lustre: server umount lustre-MDT0000 complete [ 6295.012843] LustreError: 157086:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6295.054776] LustreError: 157086:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 6297.521385] LustreError: 157069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002108 with bad export cookie 12323032830144860707 [ 6297.533369] LustreError: 157069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6297.536592] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6297.892718] Lustre: server umount lustre-MDT0001 complete [ 6301.569896] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 10:48:31 (1790002111) [ 6303.507519] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6305.654888] Lustre: DEBUG MARKER: == conf-sanity test 138: MDT mount failure must not LBUG destroying recovery exports ========================================================== 10:48:35 (1790002115) [ 6324.563812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6324.930332] LustreError: 162017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6325.001418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6329.936707] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6330.340770] LustreError: 162018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6335.461393] LustreError: 162017:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6338.719904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6343.762269] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6352.208926] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6357.465135] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6362.904202] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6363.187183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6363.192135] Lustre: Skipped 1 previous similar message [ 6368.647644] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6375.839331] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6381.276080] Lustre: *** cfs_fail_loc=718, val=0*** [ 6381.724375] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [ 6381.729632] Lustre: Skipped 1 previous similar message [ 6383.688821] Lustre: server umount lustre-MDT0000 complete [ 6385.128275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6385.144889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6385.179573] LustreError: 162746:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6387.540096] LustreError: 161998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002198 with bad export cookie 12323032830144861589 [ 6387.540497] Lustre: *** cfs_fail_loc=718, val=0*** [ 6387.551943] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6387.555312] LustreError: 161998:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6387.584745] Lustre: Skipped 3 previous similar messages [ 6387.810227] Lustre: server umount lustre-MDT0001 complete [ 6398.264091] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6406.048238] Lustre: 148511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790002201/real 1790002201] req@000000008c85e476 x1876952938034048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790002217 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6406.079881] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.089300] Lustre: Skipped 2 previous similar messages [ 6408.160399] Lustre: 148513:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790002203/real 1790002203] req@0000000046bc9644 x1876952938034176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790002219 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6411.232143] Lustre: 148512:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790002206/real 1790002206] req@00000000d11b259f x1876952938034304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790002222 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6412.258038] LustreError: 148509:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000240152a2 x1876952938035200/t0(0) o250->MGC192.168.203.144@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6412.406207] LustreError: 165357:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6412.429414] LustreError: 165357:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6412.521518] Lustre: *** cfs_fail_loc=723, val=0*** [ 6412.523792] Lustre: Skipped 1 previous similar message [ 6412.613890] LustreError: 165373:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-122) [ 6412.618324] LustreError: 165373:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.144@tcp: cfg command failed: rc = -122 [ 6412.629920] Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f [ 6412.629920] [ 6412.638331] LustreError: MGC192.168.203.144@tcp: Confguration from log lustre-MDT0000 failed from MGS -122. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info [ 6412.649549] LustreError: 165343:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-MDT0000: -122 [ 6412.655926] LustreError: 165343:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -122 [ 6412.664708] LustreError: 165343:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-MDT0000 [ 6412.674263] LustreError: 165343:0:(tgt_mount.c:116:server_deregister_mount()) lustre-MDT0000 not registered [ 6412.767575] Lustre: server umount lustre-MDT0000 complete [ 6412.773396] LustreError: 165343:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -122 [ 6413.728162] Lustre: 148511:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790002208/real 1790002208] req@00000000f44d32cc x1876952938034560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790002224 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6418.785113] Lustre: 148510:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790002213/real 1790002213] req@00000000aabf0d8b x1876952938034688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790002229 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6420.426382] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6420.552180] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6420.566575] LustreError: 162010:0:(mgc_request.c:614:do_requeue()) failed processing log: -5 [ 6425.011157] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6425.740666] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6430.180204] LustreError: 163850:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6430.204238] LustreError: 163850:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6433.468130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6433.845825] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6433.852140] Lustre: Skipped 1 previous similar message [ 6433.883765] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect [ 6437.791547] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6438.001909] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 6438.901515] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6438.926962] Lustre: lustre-MDT0001: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 6445.265256] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6446.807882] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6452.286862] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6453.839526] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6464.484849] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.487108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.496938] Lustre: Skipped 1 previous similar message [ 6469.447030] Lustre: server umount lustre-OST0000 complete [ 6474.722850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6474.722910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6474.734308] Lustre: Skipped 2 previous similar messages [ 6478.969895] Lustre: server umount lustre-MDT0000 complete [ 6479.845721] LustreError: 165786:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6479.867942] LustreError: 165786:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6482.980461] LustreError: 161997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002294 with bad export cookie 12323032830144862541 [ 6482.981293] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6482.990376] LustreError: 161997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6483.502847] Lustre: server umount lustre-MDT0001 complete [ 6494.732686] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 6497.159121] Key type lgssc unregistered [ 6497.361302] LNet: 168599:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6497.365895] LNetError: 168599:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6497.378880] LNet: Removed LNI 192.168.203.144@tcp [ 6498.071724] Key type .llcrypt unregistered [ 6498.076035] Key type ._llcrypt unregistered [ 6518.894958] Key type ._llcrypt registered [ 6518.896803] Key type .llcrypt registered [ 6519.004238] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6519.773329] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6519.887071] alg: No test for adler32 (adler32-zlib) [ 6520.899909] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 6521.075805] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 6522.720228] Key type lgssc registered [ 6523.900251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6530.646022] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6530.663614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6532.016048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6536.466509] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6542.308552] LustreError: 169931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6542.331340] LustreError: 169931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6545.089172] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6545.605389] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6549.885318] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6557.294565] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6562.611902] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6568.234161] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6568.557427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6573.518460] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6577.784864] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6581.023484] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6591.969330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6591.987191] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6592.020110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6593.506647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6593.515516] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6595.361915] Lustre: server umount lustre-MDT0000 complete [ 6598.634070] LustreError: 169928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6598.653548] LustreError: 169928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6598.708096] LustreError: 169912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002409 with bad export cookie 10713470934713506063 [ 6598.715984] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6598.732038] LustreError: 169912:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6599.313840] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [ 6599.322568] Lustre: Skipped 3 previous similar messages [ 6603.745348] LustreError: 169928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6603.745903] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6603.758263] LustreError: 169928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6603.791056] Lustre: Skipped 1 previous similar message [ 6603.803680] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6605.127045] Lustre: server umount lustre-MDT0001 complete [ 6611.575759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6625.069355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6625.371189] LustreError: 173771:0:(llog.c:1655:llog_backup()) MGC192.168.203.144@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6625.389195] Lustre: 173771:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6629.047493] LustreError: 173776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6629.082346] LustreError: 173776:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6633.442374] LustreError: 173776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6633.476966] LustreError: 173776:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6643.681291] LustreError: 173776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6643.705711] LustreError: 173776:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 6660.723755] LustreError: 173776:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6660.749682] LustreError: 173776:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 6686.176311] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6686.178819] Lustre: 173771:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.144@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6694.883867] LustreError: 171762:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6694.908896] LustreError: 171762:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 21 previous similar messages [ 6727.174978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6732.170690] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6732.277455] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6740.721900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.964245] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.112824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6750.188580] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6751.747630] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6760.287978] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6761.927488] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6767.972401] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6769.504646] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6791.145879] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.156713] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6791.159317] Lustre: Skipped 2 previous similar messages [ 6791.159599] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6791.186337] Lustre: Skipped 3 previous similar messages [ 6795.687232] Lustre: server umount lustre-MDT0000 complete [ 6796.266540] LustreError: 173796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6796.292509] LustreError: 173796:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 41 previous similar messages [ 6799.014754] LustreError: 170665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002610 with bad export cookie 10713470934713512734 [ 6799.015502] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6799.027567] LustreError: 170665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6799.323684] Lustre: server umount lustre-MDT0001 complete [ 6813.827152] Lustre: server umount lustre-OST0000 complete [ 6834.132379] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 6847.658345] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 6886.930907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6887.204478] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6887.233175] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6887.308238] Lustre: lustre-MDT0000: new disk, initializing [ 6887.407597] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6887.431756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6892.465605] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6903.270524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6903.416094] Lustre: 178879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6903.469472] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6903.475160] Lustre: Skipped 1 previous similar message [ 6903.544776] Lustre: lustre-MDT0001: new disk, initializing [ 6903.641229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6903.684761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6903.703610] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6908.610642] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6920.285430] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6926.424992] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6933.350762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6933.587633] Lustre: lustre-OST0000: new disk, initializing [ 6933.593822] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6933.601146] Lustre: 180849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6939.191508] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 6940.276275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6940.282857] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6940.415118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6950.639432] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6958.946940] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6959.389908] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6966.016503] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6966.295590] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6971.364666] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6971.365344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6971.368311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6971.371338] Lustre: Skipped 1 previous similar message [ 6982.112147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6982.336296] Lustre: server umount lustre-OST0000 complete [ 6986.721596] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6986.730189] Lustre: Skipped 1 previous similar message [ 6991.850360] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6991.861983] Lustre: Skipped 8 previous similar messages [ 6992.360737] Lustre: server umount lustre-MDT0000 complete [ 6996.069388] LustreError: 178873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002807 with bad export cookie 10713470934713529786 [ 6996.080201] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6996.085219] LustreError: 178873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6996.474993] Lustre: server umount lustre-MDT0001 complete [ 7006.230448] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 11:00:15 (1790002815) [ 7013.735056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7014.227653] LustreError: 183263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7014.311879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.320586] Lustre: Skipped 1 previous similar message [ 7018.980485] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7029.740812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7036.875138] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7046.336383] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7053.662577] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7060.635786] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7069.055621] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7077.786819] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7101.627951] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 11:01:51 (1790002911) [ 7110.123509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7110.129994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7110.165255] Lustre: Skipped 1 previous similar message [ 7112.621107] Lustre: server umount lustre-OST0000 complete [ 7117.282250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7122.861203] Lustre: server umount lustre-MDT0000 complete [ 7126.472788] LustreError: 183245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790002937 with bad export cookie 10713470934713530675 [ 7126.479927] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7126.495829] LustreError: 183245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7126.956618] Lustre: server umount lustre-MDT0001 complete [ 7136.215364] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7138.641652] Key type lgssc unregistered [ 7138.849694] LNet: 187028:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7138.861295] LNetError: 187028:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7138.884753] LNet: Removed LNI 192.168.203.144@tcp [ 7139.526906] Key type .llcrypt unregistered [ 7139.530820] Key type ._llcrypt unregistered [ 7157.164277] Key type ._llcrypt registered [ 7157.170228] Key type .llcrypt registered [ 7157.280796] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7158.247411] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7158.348640] alg: No test for adler32 (adler32-zlib) [ 7159.438353] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 7159.637588] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7161.304843] Key type lgssc registered [ 7162.444418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7169.864898] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7169.877362] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7185.440451] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 7190.560299] LustreError: 188288:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.144@tcp: failed processing log, type 4: rc = -110 [ 7195.680893] LustreError: 188288:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7195.706296] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 7195.729604] LustreError: MGC192.168.203.144@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 7195.754593] LustreError: 188288:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 7195.764476] LustreError: 188288:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 7195.775972] LustreError: 188288:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 7195.879409] Lustre: server umount lustre-OST0000 complete [ 7195.888492] LustreError: 188288:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 7206.735494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7208.157139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7214.195439] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7218.660618] LustreError: 188788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7218.673385] LustreError: 188788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7223.789427] LustreError: 188789:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7224.510625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7224.892417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7229.482253] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7237.067759] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7242.033883] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7249.482704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7249.838750] LustreError: 190603:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 7249.846542] Lustre: 190603:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.144@tcp: invalid local config log lustre-OST0000: rc = -22 [ 7249.999228] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7257.596819] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7266.934705] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7269.613455] LustreError: lustre-MDT0000-lwp-MDT0001: operation fld_query to node 0@lo failed: rc = -107 [ 7269.625788] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7269.644771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7270.888202] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7270.896840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7270.912245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.980039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7274.984157] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.997484] Lustre: Skipped 1 previous similar message [ 7275.843889] Lustre: server umount lustre-MDT0000 complete [ 7279.684500] LustreError: 191364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003090 with bad export cookie 13883605446588753397 [ 7279.692567] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7279.808256] LustreError: 189518:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000916f1534 x1876954154301312/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7279.929611] Lustre: server umount lustre-MDT0001 complete [ 7294.550036] Lustre: server umount lustre-OST0000 complete [ 7304.602766] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7314.741976] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7351.391194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.783508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7351.825702] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7351.921058] Lustre: lustre-MDT0000: new disk, initializing [ 7352.007147] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7352.044702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7357.021710] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7368.876252] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7368.980681] Lustre: 194130:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7369.033873] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7369.038024] Lustre: Skipped 1 previous similar message [ 7369.110811] Lustre: lustre-MDT0001: new disk, initializing [ 7369.221223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7369.257433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7369.266959] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7373.893076] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7383.173084] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7389.078636] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7396.361590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7396.687814] Lustre: lustre-OST0000: new disk, initializing [ 7396.693260] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7396.710217] Lustre: 196098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7396.831809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7403.000548] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7406.238716] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7406.253710] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7406.373453] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7413.626206] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7419.055937] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7419.293759] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7424.692230] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7424.895645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7432.161363] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7432.168463] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.175692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7432.191095] Lustre: Skipped 1 previous similar message [ 7432.227828] Lustre: Skipped 2 previous similar messages [ 7433.653444] Lustre: server umount lustre-OST0000 complete [ 7440.868941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7440.888152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7440.917873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7443.802937] Lustre: server umount lustre-MDT0000 complete [ 7447.521691] LustreError: 194136:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7447.539558] LustreError: 194136:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7448.213604] LustreError: 194122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003259 with bad export cookie 13883605446588754118 [ 7448.220370] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7448.232396] LustreError: 194122:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7448.635584] Lustre: server umount lustre-MDT0001 complete [ 7458.637640] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 11:07:47 (1790003267) [ 7484.222928] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 7487.160760] Key type lgssc unregistered [ 7487.541354] LNet: 199047:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7487.551774] LNetError: 199047:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7488.615779] LNet: Removed LNI 192.168.203.144@tcp [ 7489.561853] Key type .llcrypt unregistered [ 7489.566680] Key type ._llcrypt unregistered [ 7508.915969] Key type ._llcrypt registered [ 7508.925061] Key type .llcrypt registered [ 7509.085813] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7510.088396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7510.195193] alg: No test for adler32 (adler32-zlib) [ 7511.337591] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 7511.702928] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 7513.376253] Key type lgssc registered [ 7514.565202] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7522.742650] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7522.776238] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7524.287164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7529.128111] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7534.563676] LustreError: 200283:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7534.585622] LustreError: 200283:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7537.864500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.260668] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7542.659694] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7551.159684] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7558.826959] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7569.197665] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7569.551397] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7569.556151] LustreError: 202148:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7569.576147] Lustre: 202148:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.144@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7569.776936] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7576.086652] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7586.110177] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7593.450566] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7593.464335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7593.478058] Lustre: Skipped 2 previous similar messages [ 7593.496808] Lustre: Skipped 2 previous similar messages [ 7596.381055] Lustre: server umount lustre-MDT0000 complete [ 7598.574357] LustreError: 200279:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7598.593025] LustreError: 200279:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7600.862244] LustreError: 200263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003411 with bad export cookie 3265166754340828777 [ 7600.872825] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7601.394334] Lustre: server umount lustre-MDT0001 complete [ 7616.824776] Lustre: server umount lustre-OST0000 complete [ 7630.530378] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 7643.130465] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7681.176634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7681.495420] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7681.524795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7681.631208] Lustre: lustre-MDT0000: new disk, initializing [ 7681.744784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7681.778339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7686.085580] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7697.320514] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7697.490811] Lustre: 205674:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7697.540499] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7697.542418] Lustre: Skipped 1 previous similar message [ 7697.544498] LustreError: 206457:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7697.552266] LustreError: 206457:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7697.556115] Lustre: 206457:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7697.575416] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7697.583405] Lustre: Skipped 1 previous similar message [ 7697.691321] Lustre: lustre-MDT0001: new disk, initializing [ 7697.820432] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7697.869629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7697.884171] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7704.296571] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7717.042710] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7723.401628] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7730.010338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7730.304445] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7730.307688] Lustre: Skipped 1 previous similar message [ 7730.313369] LustreError: 207621:0:(llog.c:1696:llog_backup()) MGC192.168.203.144@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7730.321917] LustreError: 207621:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7730.329433] Lustre: 207621:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.144@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7730.335421] Lustre: 207621:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7730.365668] Lustre: lustre-OST0000: new disk, initializing [ 7730.369446] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7730.374543] Lustre: 207644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7730.440120] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7736.937511] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7740.018070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7740.038847] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7740.171863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7747.012170] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7752.156730] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7752.406721] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7757.404909] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7757.607707] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7760.867094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7760.867510] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7760.879706] LustreError: Skipped 1 previous similar message [ 7760.880783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7760.925799] Lustre: Skipped 1 previous similar message [ 7765.609557] Lustre: server umount lustre-OST0000 complete [ 7771.108317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7771.119117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7771.121627] Lustre: Skipped 1 previous similar message [ 7771.124104] Lustre: Skipped 1 previous similar message [ 7775.981671] Lustre: server umount lustre-MDT0000 complete [ 7776.235936] LustreError: 206807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7776.258374] LustreError: 206807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7779.797923] LustreError: 205667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003590 with bad export cookie 3265166754340829498 [ 7779.807109] LustreError: 205667:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 7779.812508] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7780.189415] Lustre: server umount lustre-MDT0001 complete [ 7789.814356] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 11:13:19 (1790003599) [ 7807.707652] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 7816.601836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7816.909484] LustreError: 210772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7816.951256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7821.131716] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7822.315211] LustreError: 210773:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7829.040808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7833.622325] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7836.772973] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7842.709647] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7849.080782] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7857.114965] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7857.323525] Lustre: lustre-OST0001: new disk, initializing [ 7857.328289] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7857.334965] Lustre: 213084:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7857.423888] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7857.429468] Lustre: Skipped 2 previous similar messages [ 7863.492594] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7863.940294] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7863.957268] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7864.101311] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7874.167572] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7886.588627] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7888.548962] Lustre: DEBUG MARKER: ADD OST3 [ 7893.512261] loop: module loaded [ 7893.573446] loop0: detected capacity change from 0 to 204800000 [ 7893.834604] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7893.855697] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7893.881997] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7899.337533] loop0: detected capacity change from 0 to 204800000 [ 7900.815305] Lustre: DEBUG MARKER: STOP OST3 [ 7905.969456] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7906.413624] Lustre: lustre-OST0003: new disk, initializing [ 7906.431696] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7906.447266] Lustre: 215072:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7907.806911] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7907.966864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7907.975950] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7908.025765] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7909.857465] LustreError: 215176:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7911.332859] Lustre: Failing over lustre-OST0003 [ 7911.464407] Lustre: server umount lustre-OST0003 complete [ 7912.965881] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7912.968919] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7912.989222] Lustre: Skipped 1 previous similar message [ 7912.997202] LustreError: 213468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7913.045940] LustreError: 213468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7914.505672] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7924.244475] Lustre: DEBUG MARKER: START OST3 again [ 7931.050900] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7931.325502] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7931.330948] Lustre: Skipped 1 previous similar message [ 7931.343836] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7932.435661] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7932.681030] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7932.681481] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7936.922793] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 7957.728231] Lustre: Failing over lustre-OST0003 [ 7957.890661] Lustre: server umount lustre-OST0003 complete [ 7958.497267] LustreError: lustre-OST0003-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7958.510329] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7958.522876] LustreError: 212263:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7958.529678] Lustre: Skipped 1 previous similar message [ 7958.560523] LustreError: 212263:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 7977.450912] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7977.472177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7977.478115] Lustre: Skipped 2 previous similar messages [ 7977.499574] Lustre: Skipped 4 previous similar messages [ 7981.977482] Lustre: server umount lustre-MDT0000 complete [ 7985.747875] LustreError: 213082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003796 with bad export cookie 3265166754340830380 [ 7985.759150] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7985.769655] LustreError: 213082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7987.690476] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 7990.754238] LustreError: 210767:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7990.790394] LustreError: 210767:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 7992.166405] Lustre: server umount lustre-MDT0001 complete [ 7996.515509] Lustre: server umount lustre-OST0000 complete [ 8000.483366] Lustre: server umount lustre-OST0001 complete [ 8006.385638] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8014.381213] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8053.685105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8053.857260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8053.905732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8053.985152] Lustre: lustre-MDT0000: new disk, initializing [ 8054.069708] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8059.897981] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8072.064178] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.144797] Lustre: 220541:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8072.189098] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8072.194402] Lustre: Skipped 1 previous similar message [ 8072.247752] Lustre: lustre-MDT0001: new disk, initializing [ 8072.419809] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8072.436581] Lustre: Skipped 1 previous similar message [ 8072.496783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8072.511981] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8078.271512] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8089.218307] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8095.822790] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8103.555808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8103.988324] Lustre: 222509:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8106.030465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8106.043433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8106.134285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8111.456862] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8121.853021] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8126.495786] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8126.680849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8131.535946] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8131.708751] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8137.195180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8137.197783] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8137.230232] Lustre: Skipped 3 previous similar messages [ 8137.232423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8137.250943] LustreError: Skipped 1 previous similar message [ 8137.284618] Lustre: Skipped 4 previous similar messages [ 8140.510643] Lustre: server umount lustre-OST0000 complete [ 8144.358660] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8144.370292] LustreError: Skipped 1 previous similar message [ 8150.757210] Lustre: server umount lustre-MDT0000 complete [ 8152.546982] LustreError: 220554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8152.566230] LustreError: 220554:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8155.065668] LustreError: 220536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790003966 with bad export cookie 3265166754340832669 [ 8155.073691] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8155.084879] LustreError: 220536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8155.527764] Lustre: server umount lustre-MDT0001 complete [ 8159.647991] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 11:19:29 (1790003969) [ 8177.879823] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8193.293495] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8240.530785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8240.780908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8240.804801] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8240.808320] Lustre: Skipped 1 previous similar message [ 8240.885383] Lustre: lustre-MDT0000: new disk, initializing [ 8240.895716] Lustre: Skipped 1 previous similar message [ 8241.044860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8246.628701] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8262.119964] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8262.307601] Lustre: 227145:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8262.800994] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8269.445760] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8284.415310] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8294.101591] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8304.665522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8304.978479] Lustre: lustre-OST0000: new disk, initializing [ 8304.984481] Lustre: Skipped 1 previous similar message [ 8304.988377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8304.991103] Lustre: Skipped 2 previous similar messages [ 8305.002390] Lustre: 229118:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8306.805268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8307.039363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8314.386416] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8325.146291] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8329.688504] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8329.995865] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8336.268109] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8336.434867] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8343.011497] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.015176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8343.044914] Lustre: Skipped 3 previous similar messages [ 8343.059464] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8343.074833] Lustre: Skipped 4 previous similar messages [ 8345.656241] Lustre: server umount lustre-OST0000 complete [ 8357.055278] Lustre: server umount lustre-MDT0000 complete [ 8358.369206] LustreError: 227153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8358.381151] LustreError: 227153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8360.080201] LustreError: 227940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004171 with bad export cookie 3265166754340833551 [ 8360.083125] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8360.087104] LustreError: 227940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8360.385957] Lustre: server umount lustre-MDT0001 complete [ 8369.403866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8370.336378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8370.353925] Lustre: Skipped 4 previous similar messages [ 8375.238065] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8385.926784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.241987] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8396.809438] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8404.283537] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8413.279547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8422.644318] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8431.425970] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8441.828236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8441.828548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8441.829549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8441.829554] Lustre: Skipped 2 previous similar messages [ 8441.835574] LustreError: Skipped 1 previous similar message [ 8441.894556] Lustre: Skipped 4 previous similar messages [ 8447.821738] Lustre: server umount lustre-OST0000 complete [ 8457.634763] Lustre: server umount lustre-MDT0000 complete [ 8461.164212] LustreError: 231315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004272 with bad export cookie 3265166754340834433 [ 8461.171504] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8461.175490] LustreError: 231315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8461.456624] Lustre: server umount lustre-MDT0001 complete [ 8572.797559] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 8575.608802] Key type lgssc unregistered [ 8576.024463] LNet: 235001:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8576.037752] LNetError: 235001:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8576.058733] LNet: Removed LNI 192.168.203.144@tcp [ 8577.177175] Key type .llcrypt unregistered [ 8577.183596] Key type ._llcrypt unregistered [ 8613.657294] Key type ._llcrypt registered [ 8613.662733] Key type .llcrypt registered [ 8613.851356] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 8632.293398] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8633.474224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8633.583295] alg: No test for adler32 (adler32-zlib) [ 8634.700498] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 8634.930739] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 8636.632441] Key type lgssc registered [ 8637.976129] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8685.499406] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8685.546969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8686.839752] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8686.920474] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8686.977188] Lustre: lustre-MDT0000: new disk, initializing [ 8687.035504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8687.048581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8691.791948] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8703.062330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8703.256399] Lustre: 238375:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8703.413674] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8703.419099] Lustre: Skipped 1 previous similar message [ 8703.553824] Lustre: lustre-MDT0001: new disk, initializing [ 8703.800600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8703.898056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8703.937855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8711.527596] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8724.546598] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8730.588142] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8738.065834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8738.393271] Lustre: lustre-OST0000: new disk, initializing [ 8738.399340] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8738.412517] Lustre: 240348:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8738.511580] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8744.413968] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8745.038943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8745.055209] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8745.181535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8754.042496] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8760.427166] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8760.611616] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8768.257660] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8768.526769] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8771.048495] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8771.056376] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8771.066893] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8771.078374] Lustre: Skipped 1 previous similar message [ 8776.163763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8776.183395] Lustre: Skipped 1 previous similar message [ 8781.286490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8781.302530] Lustre: Skipped 1 previous similar message [ 8785.376166] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8785.875457] Lustre: server umount lustre-OST0000 complete [ 8791.009111] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8791.019153] LustreError: Skipped 1 previous similar message [ 8791.028908] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8791.049225] Lustre: Skipped 1 previous similar message [ 8791.056638] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8796.644397] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8796.655481] Lustre: Skipped 2 previous similar messages [ 8805.344276] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8805.881501] Lustre: server umount lustre-MDT0000 complete [ 8809.768188] LustreError: 238367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004620 with bad export cookie 13503159809811394277 [ 8809.769772] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8809.779686] LustreError: 238367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8809.992976] Lustre: server umount lustre-MDT0001 complete [ 8835.043226] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 8847.975237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8848.880271] LustreError: 243306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8848.902770] LustreError: 243306:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8848.955576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8853.984725] LustreError: 243307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8854.507881] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8859.128174] LustreError: 243306:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8864.260199] LustreError: 243307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8866.189463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8866.781648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8871.660668] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8874.666562] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8882.786447] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8883.094807] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8892.589579] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8903.861194] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8904.104727] Lustre: lustre-OST0001: new disk, initializing [ 8904.110854] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8904.127380] Lustre: 245614:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8904.322284] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8911.523800] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8911.541603] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8911.635398] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8911.697349] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 8925.323509] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8932.841564] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8948.915078] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 11:32:38 (1790004758) [ 8952.812173] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.819923] Lustre: Skipped 3 previous similar messages [ 8952.824838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8952.836271] Lustre: Skipped 5 previous similar messages [ 8957.778757] Lustre: server umount lustre-MDT0000 complete [ 8957.935497] LustreError: 244810:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.979104] LustreError: 244810:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8962.422743] LustreError: 243286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004773 with bad export cookie 13503159809811395159 [ 8962.431037] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8962.437027] LustreError: 243286:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8963.022229] Lustre: server umount lustre-MDT0001 complete [ 8977.717830] Lustre: server umount lustre-OST0000 complete [ 8979.553092] Lustre: 236587:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790004774/real 1790004774] req@ffff9fb7f6186d80 x1876955702088960/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790004790 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8979.615795] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8979.636747] Lustre: Skipped 1 previous similar message [ 8982.381229] Lustre: server umount lustre-OST0001 complete [ 8990.191489] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9002.330126] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9051.446076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9051.724361] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9051.749427] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9051.802693] Lustre: lustre-MDT0000: new disk, initializing [ 9051.879234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9051.895114] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9055.772371] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9071.938770] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9072.330914] Lustre: 249566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9072.488096] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9072.495743] Lustre: Skipped 1 previous similar message [ 9072.722241] Lustre: lustre-MDT0001: new disk, initializing [ 9072.973225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9073.000169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9080.793474] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9090.650974] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9096.343328] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9103.487765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9103.785720] Lustre: lustre-OST0000: new disk, initializing [ 9103.789725] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9103.799834] Lustre: 251537:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9105.645790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9105.683596] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9105.816298] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9110.712537] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9120.026721] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9124.678365] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9124.911397] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9129.728795] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9130.015214] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9136.613891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9136.645761] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9136.653993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9138.675757] Lustre: server umount lustre-OST0000 complete [ 9144.804701] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9144.810705] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9144.824759] Lustre: Skipped 1 previous similar message [ 9150.177603] Lustre: server umount lustre-MDT0000 complete [ 9151.986024] LustreError: 249580:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9152.015337] LustreError: 249580:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9154.829278] LustreError: 249561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790004965 with bad export cookie 13503159809811397294 [ 9154.865635] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9154.865982] LustreError: 249561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9155.743874] Lustre: server umount lustre-MDT0001 complete [ 9167.352039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9167.838655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9167.843953] Lustre: Skipped 2 previous similar messages [ 9172.964407] LustreError: 253756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9172.991020] LustreError: 253756:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9173.079349] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9185.038834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9190.753733] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9200.232390] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9207.078288] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9215.027462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9222.957090] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9232.699473] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9381.859055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9381.862443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9381.894449] Lustre: Skipped 1 previous similar message [ 9381.909115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9381.921480] Lustre: Skipped 5 previous similar messages [ 9385.585753] Lustre: server umount lustre-OST0000 complete [ 9397.425030] Lustre: server umount lustre-MDT0000 complete [ 9401.763977] LustreError: 254489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005212 with bad export cookie 13503159809811398176 [ 9401.768599] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9401.775562] LustreError: 254489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9402.065839] Lustre: server umount lustre-MDT0001 complete [ 9505.576921] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [ 9510.263487] Key type lgssc unregistered [ 9510.788515] LNet: 257363:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9510.802252] LNetError: 257363:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9510.833376] LNet: Removed LNI 192.168.203.144@tcp [ 9512.330182] Key type .llcrypt unregistered [ 9512.334614] Key type ._llcrypt unregistered [ 9537.963588] Key type ._llcrypt registered [ 9537.966903] Key type .llcrypt registered [ 9538.025270] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9555.778171] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9556.813272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9556.831135] alg: No test for adler32 (adler32-zlib) [ 9558.055321] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [ 9558.382220] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [ 9560.056181] Key type lgssc registered [ 9561.675380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9615.786726] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9631.900449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9631.921920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9633.531397] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9633.647471] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9633.980195] Lustre: lustre-MDT0000: new disk, initializing [ 9634.229152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9634.273554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9641.499250] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9655.918530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9656.051694] Lustre: 261452:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9656.097908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9656.101609] Lustre: Skipped 1 previous similar message [ 9656.160745] Lustre: lustre-MDT0001: new disk, initializing [ 9656.233810] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9656.265361] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9656.274789] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9661.113442] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9666.295231] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9673.212974] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9673.497232] Lustre: lustre-OST0000: new disk, initializing [ 9673.504172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9673.523784] Lustre: 263086:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9673.683203] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9676.952422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9676.966974] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9677.194784] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9680.707682] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9691.656794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9691.880744] Lustre: lustre-OST0001: new disk, initializing [ 9691.884561] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9691.890816] Lustre: 263960:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9691.981983] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9698.392555] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9699.956553] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9699.968807] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9700.075919] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9710.259936] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9716.293948] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9731.046410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9731.059885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9731.074523] Lustre: Skipped 3 previous similar messages [ 9731.085642] Lustre: Skipped 2 previous similar messages [ 9735.083484] Lustre: server umount lustre-MDT0000 complete [ 9739.238616] LustreError: 261443:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005550 with bad export cookie 6004202528998803849 [ 9739.257996] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9739.945674] Lustre: server umount lustre-MDT0001 complete [ 9753.992625] Lustre: server umount lustre-OST0000 complete [ 9757.474131] Lustre: 258947:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790005552/real 1790005552] req@ffff9fb6c8d3b100 x1876956669882752/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790005568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9757.500559] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9758.016837] Lustre: server umount lustre-OST0001 complete [ 9774.512077] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9784.087291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9793.718192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9804.237295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9824.867804] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [ 9835.076440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9835.093483] Lustre: lustre-MDT0000: reset Object Index mappings [ 9835.518029] LustreError: 268970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9835.545568] LustreError: 268970:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9835.607167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9840.206630] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9840.612322] LustreError: 268971:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9845.734977] LustreError: 268970:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9849.256081] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9849.277225] Lustre: lustre-MDT0001: reset Object Index mappings [ 9849.618729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9854.027630] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9857.129235] Lustre: 270112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9864.859372] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9871.206918] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9874.342220] LustreError: 270464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9879.529806] LustreError: 270467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9880.495438] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9880.607836] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9880.840347] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9880.843528] Lustre: Skipped 1 previous similar message [ 9886.215360] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9886.526774] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [ 9895.006943] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9899.463666] Lustre: 271983:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9905.693485] Lustre: server umount lustre-MDT0000 complete [ 9906.145972] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9906.165938] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9906.179168] LustreError: 269703:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9906.200416] LustreError: 269703:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9909.643558] LustreError: 268952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005720 with bad export cookie 6004202528998810639 [ 9909.648718] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9909.661363] LustreError: 268952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9910.061658] Lustre: server umount lustre-MDT0001 complete [ 9924.485347] Lustre: server umount lustre-OST0000 complete [ 9927.650429] Lustre: 258946:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790005722/real 1790005722] req@ffff9fb7fde0ea00 x1876956669929472/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790005738 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9927.701076] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9927.721900] Lustre: Skipped 3 previous similar messages [ 9928.874424] Lustre: server umount lustre-OST0001 complete [ 9951.877701] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 11:49:21 (1790005761) [ 9970.251420] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [ 9984.948530] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10029.206878] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10029.476803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10029.539820] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10029.645933] Lustre: lustre-MDT0000: new disk, initializing [10029.730393] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10029.748152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10035.762346] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10048.827500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10048.984817] Lustre: 276259:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10049.058555] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10049.073414] Lustre: Skipped 1 previous similar message [10049.270977] Lustre: lustre-MDT0001: new disk, initializing [10049.596463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10049.617797] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10055.003939] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10065.391701] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10071.915565] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10080.179616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10080.526481] Lustre: lustre-OST0000: new disk, initializing [10080.530149] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10080.535975] Lustre: 278228:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10082.098942] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10082.120439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10082.281888] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10086.655153] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10100.145957] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10106.575593] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10106.882168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10115.294845] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10115.701893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10121.189721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10121.220330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10121.245341] Lustre: Skipped 1 previous similar message [10124.919933] Lustre: server umount lustre-OST0000 complete [10131.429398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10131.453884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10131.464643] Lustre: Skipped 1 previous similar message [10131.495909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10131.508153] Lustre: Skipped 1 previous similar message [10136.547951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10136.552941] Lustre: Skipped 1 previous similar message [10137.448304] Lustre: server umount lustre-MDT0000 complete [10141.283512] LustreError: 276253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790005952 with bad export cookie 6004202528998812732 [10141.292706] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10141.304783] LustreError: 276253:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10141.606874] Lustre: server umount lustre-MDT0001 complete [10163.964197] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10177.936320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10178.257831] LustreError: 281165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10178.277334] LustreError: 281165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10178.390275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10178.399299] Lustre: Skipped 2 previous similar messages [10182.432732] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10192.208885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10196.871075] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10199.830830] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [10206.749239] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10213.421587] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10221.617610] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10221.812666] Lustre: lustre-OST0001: new disk, initializing [10221.817337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10221.836905] Lustre: 283473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10223.046342] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10223.088095] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10223.344524] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10229.629953] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10240.877180] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10246.519859] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10252.982847] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [10253.270167] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [10264.548849] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10264.566058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10264.579859] Lustre: Skipped 4 previous similar messages [10264.587749] Lustre: Skipped 2 previous similar messages [10264.604407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10270.841971] Lustre: server umount lustre-MDT0000 complete [10274.748175] LustreError: 281901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006085 with bad export cookie 6004202528998813614 [10274.749339] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10274.756209] LustreError: 281901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10274.785886] LustreError: 284547:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10274.799980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10274.803485] LustreError: 284547:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10274.831029] Lustre: Skipped 6 previous similar messages [10275.052150] Lustre: server umount lustre-MDT0001 complete [10281.985628] Lustre: server umount lustre-OST0000 complete [10286.837396] Lustre: server umount lustre-OST0001 complete [10312.623435] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10322.864874] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10323.265427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10323.271217] Lustre: Skipped 3 previous similar messages [10328.288307] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10337.221460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10342.016200] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10345.018876] Lustre: 287958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [10352.284676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10362.178967] LustreError: 288311:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10362.216940] LustreError: 288311:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10363.415603] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10368.590348] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [10376.223982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10381.839122] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [10384.199982] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10391.582549] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10395.126587] Lustre: 289834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [10402.308856] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [10402.342203] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [10421.671673] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 11:57:10 (1790006230) [10438.117292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10438.127937] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10438.141297] Lustre: Skipped 5 previous similar messages [10438.155964] Lustre: Skipped 3 previous similar messages [10444.050149] Lustre: server umount lustre-MDT0000 complete [10447.929313] LustreError: 286798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006259 with bad export cookie 6004202528998820285 [10447.930672] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10447.950453] LustreError: 286798:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10454.668886] Lustre: server umount lustre-MDT0001 complete [10465.007552] Lustre: server umount lustre-OST0000 complete [10470.007505] Lustre: server umount lustre-OST0001 complete [10476.784457] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [10487.938210] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10530.208408] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10540.820140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10540.983347] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10541.004332] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10541.069297] Lustre: lustre-MDT0000: new disk, initializing [10541.148805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10546.912443] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10560.510111] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10560.653634] Lustre: 294366:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10560.954995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10565.763419] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10577.700872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10578.049096] Lustre: 296001:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10580.197089] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10586.580802] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10600.669150] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10600.958719] Lustre: 296872:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10601.094351] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10601.105240] Lustre: Skipped 6 previous similar messages [10602.279908] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10602.289307] Lustre: Skipped 1 previous similar message [10602.389273] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10606.249985] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10615.770976] Lustre: DEBUG MARKER: Using TIMEOUT=20 [10619.871716] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [10619.877951] Lustre: Skipped 1 previous similar message [10633.690629] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 12:00:43 (1790006443) [10637.795819] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10637.807718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10637.825712] Lustre: Skipped 2 previous similar messages [10637.847023] LustreError: 294377:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10637.859585] LustreError: 294377:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [10637.930542] Lustre: server umount lustre-MDT0000 complete [10642.227386] LustreError: 295999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006453 with bad export cookie 6004202528998827978 [10642.239415] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10642.243751] LustreError: 295999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10642.815793] Lustre: server umount lustre-MDT0001 complete [10657.784399] Lustre: server umount lustre-OST0000 complete [10659.746520] Lustre: 258947:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1790006454/real 1790006454] req@ffff9fb7ff369c00 x1876956670199552/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1790006470 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [10661.306912] Lustre: server umount lustre-OST0001 complete [10669.328708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10674.451538] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10684.128070] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10689.132238] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10698.023216] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10704.190306] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10714.701079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10723.465902] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10730.479204] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10758.054828] loop1: detected capacity change from 0 to 204800000 [10758.273410] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10758.290411] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10758.329937] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10763.551744] loop1: detected capacity change from 0 to 204800000 [10770.387945] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10770.549663] LustreError: 299118:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs25110: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10770.564706] LustreError: 299118:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs25110 [10770.569881] LustreError: 299118:0:(mgs_handler.c:517:mgs_target_reg()) fs25110-MDT0001: Error checking target index: rc = -13 [10770.581136] LustreError: fs25110-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10770.595669] LustreError: 302876:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10770.604656] LustreError: 302876:0:(tgt_mount.c:2128:server_put_super()) no obd fs25110-MDT0001 [10770.617152] LustreError: 302876:0:(tgt_mount.c:116:server_deregister_mount()) fs25110-MDT0001 not registered [10770.650613] Lustre: server umount fs25110-MDT0001 complete [10770.657402] LustreError: 302876:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10779.642200] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10779.769467] Lustre: Setting parameter fs25110-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs25110-MDT0001 [10779.803752] Lustre: srv-fs25110-MDT0001: No data found on store. Initialize space. [10779.808978] Lustre: Skipped 4 previous similar messages [10779.884871] Lustre: fs25110-MDT0001: new disk, initializing [10779.889370] Lustre: Skipped 3 previous similar messages [10779.984959] Lustre: srv-fs25110-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10779.989743] Lustre: 303342:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25110-MDT0001-mdtlov: can't create new llog: rc = -115 [10779.994700] LustreError: 303342:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25110-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10785.474536] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10793.609391] Lustre: server umount fs25110-MDT0001 complete [10808.866895] loop1: detected capacity change from 0 to 204800000 [10810.863520] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10815.479534] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.166842] Lustre: 304911:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25110-MDT0001-mdtlov: can't create new llog: rc = -115 [10816.177870] LustreError: 304911:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25110-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10821.017601] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10826.772710] Lustre: server umount fs25110-MDT0001 complete [10841.571576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10841.577177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10841.580290] Lustre: Skipped 6 previous similar messages [10841.582814] Lustre: Skipped 9 previous similar messages [10853.345347] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10853.621751] Lustre: server umount lustre-MDT0000 complete [10863.572856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10863.768252] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10869.239363] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10870.597572] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10879.978760] loop1: detected capacity change from 0 to 204800000 [10886.734628] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.418737] Lustre: 307359:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs25110-MDT0001-mdtlov: can't create new llog: rc = -115 [10887.441040] LustreError: 307359:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs25110-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10893.002420] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [10894.828532] LustreError: 300959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10894.848483] LustreError: 300959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 106 previous similar messages [10898.345737] Lustre: server umount fs25110-MDT0001 complete [10925.539587] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10925.557821] LustreError: Skipped 1 previous similar message [10925.564238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10925.581888] Lustre: Skipped 10 previous similar messages [10939.360682] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10939.666478] Lustre: server umount lustre-OST0000 complete [10952.249388] Lustre: server umount lustre-MDT0000 complete [10956.581392] LustreError: 299111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790006767 with bad export cookie 6004202528998830659 [10956.583119] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10956.591921] LustreError: 299111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10956.873137] Lustre: server umount lustre-MDT0001 complete [10970.597379] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [10974.356731] Key type lgssc unregistered [10974.624068] LNet: 309705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10974.640656] LNetError: 309705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10974.663779] LNet: Removed LNI 192.168.203.144@tcp [10975.983329] Key type .llcrypt unregistered [10975.991586] Key type ._llcrypt unregistered [10996.596730] Key type ._llcrypt registered [10996.599597] Key type .llcrypt registered [10996.692664] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [10997.978155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10998.073970] alg: No test for adler32 (adler32-zlib) [10999.093310] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [10999.432587] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11001.201035] Key type lgssc registered [11002.519918] Lustre: Echo OBD driver; http://www.lustre.org/ [11010.009430] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11010.034544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11011.509802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11016.305756] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11021.805558] LustreError: 310994:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11021.828976] LustreError: 310994:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11025.460091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11026.197325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11032.350277] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11040.143672] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11045.608954] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11052.665084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11052.997288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11056.051931] LustreError: 312823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11059.993887] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11061.217728] LustreError: 312823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11066.340751] LustreError: 312822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11068.186646] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11071.056661] LustreError: 313247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.334652] LustreError: 313247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11081.356173] LustreError: 313247:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11089.581349] loop1: detected capacity change from 0 to 204800000 [11089.756679] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11089.770736] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11089.784573] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11095.206798] loop1: detected capacity change from 0 to 204800000 [11101.087795] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11101.258146] Lustre: fs1638-OST0000: new disk, initializing [11101.261735] Lustre: srv-fs1638-OST0000: No data found on store. Initialize space. [11101.267201] Lustre: 314730:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11101.354375] Lustre: fs1638-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11101.825974] LustreError: 312822:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11101.851428] LustreError: 312822:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [11107.229506] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11116.478170] Lustre: server umount fs1638-OST0000 complete [11131.242467] loop1: detected capacity change from 0 to 204800000 [11137.131412] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11137.475192] Lustre: fs1638-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11137.648472] LustreError: 312823:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11137.662387] LustreError: 312823:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [11138.029367] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11143.525996] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11152.132070] loop2: detected capacity change from 0 to 204800000 [11152.346052] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11152.370625] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11152.390854] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11153.394118] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [11157.469287] loop2: detected capacity change from 0 to 204800000 [11163.169303] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11163.258140] LustreError: fs1638-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [11163.265927] LustreError: fs1638-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [11163.271615] LustreError: 317774:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [11163.276257] LustreError: 317774:0:(tgt_mount.c:2128:server_put_super()) no obd fs1638-OST0001 [11163.287374] LustreError: 317774:0:(tgt_mount.c:116:server_deregister_mount()) fs1638-OST0001 not registered [11163.326216] Lustre: server umount fs1638-OST0001 complete [11163.329901] LustreError: 317774:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [11173.611548] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11173.721323] Lustre: fs1638-OST0001: new disk, initializing [11173.729939] Lustre: srv-fs1638-OST0001: No data found on store. Initialize space. [11173.734780] Lustre: 318325:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11173.817063] Lustre: fs1638-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11179.864648] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11189.559424] Lustre: server umount fs1638-OST0000 complete [11202.272826] Lustre: server umount fs1638-OST0001 complete [11204.220087] LustreError: 313247:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11204.240589] LustreError: 313247:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [11216.075234] loop1: detected capacity change from 0 to 204800000 [11222.219433] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11222.378079] Lustre: fs1638-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11228.744354] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11236.723151] loop2: detected capacity change from 0 to 204800000 [11245.638316] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11254.138615] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11266.486916] Lustre: server umount fs1638-OST0000 complete [11280.365989] Lustre: server umount fs1638-OST0001 complete [11291.645373] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [11296.745127] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11296.773805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11296.774262] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11300.557150] Lustre: server umount lustre-OST0000 complete [11306.977893] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11306.982158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11307.004357] Lustre: Skipped 2 previous similar messages [11307.017582] Lustre: Skipped 2 previous similar messages [11312.102215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11312.113891] Lustre: Skipped 1 previous similar message [11312.490781] Lustre: server umount lustre-MDT0000 complete [11317.127725] LustreError: 310975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007128 with bad export cookie 1180030202092844817 [11317.142892] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11317.149486] LustreError: 310975:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11317.941491] Lustre: server umount lustre-MDT0001 complete [11333.372933] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11336.611785] Key type lgssc unregistered [11337.065820] LNet: 324250:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11337.075245] LNetError: 324250:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11337.093470] LNet: Removed LNI 192.168.203.144@tcp [11338.235935] Key type .llcrypt unregistered [11338.238431] Key type ._llcrypt unregistered [11377.283335] Key type ._llcrypt registered [11377.287705] Key type .llcrypt registered [11377.372719] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11393.381889] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11394.272309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11394.339253] alg: No test for adler32 (adler32-zlib) [11395.400561] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [11395.636539] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11397.384230] Key type lgssc registered [11398.787103] Lustre: Echo OBD driver; http://www.lustre.org/ [11436.676676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11436.707692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11438.116603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11438.170345] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11438.374203] Lustre: lustre-MDT0000: new disk, initializing [11438.531754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11438.549919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11444.570342] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11457.875758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11458.115142] Lustre: 328127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11458.247774] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11458.270176] Lustre: Skipped 1 previous similar message [11458.431595] Lustre: lustre-MDT0001: new disk, initializing [11458.572453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11458.622406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11458.636170] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11463.762165] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11473.160955] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11478.743075] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11486.113856] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11486.301692] Lustre: lustre-OST0000: new disk, initializing [11486.306371] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11486.311926] Lustre: 330100:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11486.380881] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11493.462277] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11495.038432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11495.050691] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11495.184514] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11504.655053] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11506.744215] Lustre: Mounted lustre-client - version 2.17.58_63_g3df9749 [11507.798524] Lustre: Unmounted lustre-client [11520.993954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11521.009557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11521.024533] Lustre: Skipped 1 previous similar message [11521.025467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11521.043034] Lustre: Skipped 1 previous similar message [11522.021514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11522.027465] Lustre: Skipped 1 previous similar message [11527.137609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11532.256159] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11532.261330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11532.269818] Lustre: Skipped 2 previous similar messages [11532.522233] Lustre: server umount lustre-OST0000 complete [11537.378146] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.402775] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11537.409268] Lustre: Skipped 1 previous similar message [11543.048345] Lustre: server umount lustre-MDT0000 complete [11547.478978] LustreError: 328119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007358 with bad export cookie 3018958479618364048 [11547.487981] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11547.501774] LustreError: 328119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11547.782976] Lustre: server umount lustre-MDT0001 complete [11561.210141] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11564.006287] Key type lgssc unregistered [11564.640148] LNet: 1 local NIs in recovery (showing 1): 192.168.203.144@tcp [11565.408596] LNet: 332261:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11565.415230] LNetError: 332261:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11638.187267] LNet: Removed LNI 192.168.203.144@tcp [11638.992714] Key type .llcrypt unregistered [11638.994565] Key type ._llcrypt unregistered [11656.987526] Key type ._llcrypt registered [11656.997700] Key type .llcrypt registered [11657.145882] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [11672.698849] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11673.828045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11673.884235] alg: No test for adler32 (adler32-zlib) [11675.074673] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [11675.371793] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11677.080185] Key type lgssc registered [11678.539157] Lustre: Echo OBD driver; http://www.lustre.org/ [11711.459273] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 12:18:41 (1790007521) [11717.678055] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11717.699909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11719.001099] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11719.044687] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11719.175331] Lustre: lustre-MDT0000: new disk, initializing [11719.247197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11719.279274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11724.012358] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11734.526819] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11734.673276] Lustre: 335499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11734.719889] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11734.723374] Lustre: Skipped 1 previous similar message [11734.851179] Lustre: lustre-MDT0001: new disk, initializing [11734.950602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11734.976526] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11734.986166] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11739.275444] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11749.442315] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11755.438254] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11761.747501] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11762.014889] Lustre: lustre-OST0000: new disk, initializing [11762.019085] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11762.028617] Lustre: 337466:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11762.129894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11767.918449] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11768.902038] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11768.924208] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11769.069473] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11777.969138] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11794.916397] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11794.918822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11794.935336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11794.940518] Lustre: Skipped 1 previous similar message [11794.957409] Lustre: Skipped 1 previous similar message [11798.488469] Lustre: server umount lustre-OST0000 complete [11805.157405] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11805.160625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11805.168977] Lustre: Skipped 1 previous similar message [11805.195960] Lustre: Skipped 1 previous similar message [11808.857397] Lustre: server umount lustre-MDT0000 complete [11813.614609] LustreError: 335491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007624 with bad export cookie 3607694038953802706 [11813.618854] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11813.622805] LustreError: 335491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11814.060483] Lustre: server umount lustre-MDT0001 complete [11827.846103] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [11830.526459] Key type lgssc unregistered [11830.863720] LNet: 339374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11830.882227] LNetError: 339374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11830.899425] LNet: Removed LNI 192.168.203.144@tcp [11831.963148] Key type .llcrypt unregistered [11831.964908] Key type ._llcrypt unregistered [11854.126167] Key type ._llcrypt registered [11854.128289] Key type .llcrypt registered [11854.288545] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [11855.641325] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11855.702355] alg: No test for adler32 (adler32-zlib) [11856.825735] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [11857.097394] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [11858.744215] Key type lgssc registered [11859.939793] Lustre: Echo OBD driver; http://www.lustre.org/ [11867.508748] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11867.544495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11869.092842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11873.871927] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11879.401695] LustreError: 340664:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11879.425849] LustreError: 340664:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11882.124419] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11882.474240] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11887.176791] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11895.124089] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11901.512387] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11909.309046] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11909.807330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11915.840145] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11924.151654] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11937.649846] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11939.516576] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11943.905452] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11943.915052] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11943.949908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11946.659208] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [11948.009153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11948.014292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11948.055914] Lustre: Skipped 2 previous similar messages [11951.733692] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [11956.854107] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.44@tcp (stopping) [11956.862015] Lustre: Skipped 3 previous similar messages [11958.240323] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11958.454888] Lustre: server umount lustre-MDT0000 complete [11961.970771] LustreError: 340660:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11962.338150] LustreError: 340646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007773 with bad export cookie 11253619879851688974 [11962.344708] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11962.355641] LustreError: 340646:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11963.361913] LustreError: 341668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11963.365692] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.400133] Lustre: Skipped 1 previous similar message [11967.100986] LustreError: 341668:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.44@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11967.129703] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.44@tcp (stopping) [11967.134647] Lustre: Skipped 4 previous similar messages [11968.829391] Lustre: server umount lustre-MDT0001 complete [11972.814570] Lustre: server umount lustre-OST0000 complete [11981.901365] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11986.671750] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [11989.333862] Lustre: 344847:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.144@tcp [11991.519477] Lustre: server umount MGS complete [12000.626635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12001.049535] LustreError: 345420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12001.064730] LustreError: 345420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12001.136270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12005.722255] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12011.489418] LustreError: 345421:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12011.524561] LustreError: 345421:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12014.076228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12014.644264] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12019.655301] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12028.858813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12029.427896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12036.362461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12044.592514] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12046.301509] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [12052.658397] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12054.597263] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12060.129830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12060.142419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12060.166154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12060.177108] Lustre: Skipped 1 previous similar message [12061.156828] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [12061.962116] Lustre: server umount lustre-OST0000 complete [12067.298841] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12067.318287] Lustre: Skipped 2 previous similar messages [12072.291662] Lustre: server umount lustre-MDT0000 complete [12072.419630] LustreError: 345420:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12072.442540] LustreError: 345420:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12075.855597] LustreError: 345400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790007886 with bad export cookie 11253619879851689814 [12075.857888] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12075.873075] LustreError: 345400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12076.246850] Lustre: server umount lustre-MDT0001 complete [12088.979269] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12092.358847] Key type lgssc unregistered [12092.662772] LNet: 348860:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12092.677255] LNetError: 348860:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12092.728662] LNet: Removed LNI 192.168.203.144@tcp [12093.715683] Key type .llcrypt unregistered [12093.717961] Key type ._llcrypt unregistered [12111.956638] Key type ._llcrypt registered [12111.960298] Key type .llcrypt registered [12112.094966] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12129.873465] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12131.014512] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12131.210366] alg: No test for adler32 (adler32-zlib) [12132.315309] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [12132.554358] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [12134.264333] Key type lgssc registered [12135.493869] Lustre: Echo OBD driver; http://www.lustre.org/ [12171.532627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12171.557071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12172.890396] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12172.926530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12173.023297] Lustre: lustre-MDT0000: new disk, initializing [12173.210473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12173.278802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12177.757311] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12187.805868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12187.956225] Lustre: 352040:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12188.007619] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12188.011917] Lustre: Skipped 1 previous similar message [12188.095539] Lustre: lustre-MDT0001: new disk, initializing [12188.260161] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12188.320466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12188.334325] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12192.970667] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12202.356505] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12207.920685] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12216.140234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12216.414374] Lustre: lustre-OST0000: new disk, initializing [12216.419482] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12216.424707] Lustre: 354003:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12216.495653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12222.037589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12222.051339] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12222.152303] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12223.173924] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12233.052954] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12238.988175] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12239.202669] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12244.057062] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12244.308446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12248.033929] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12248.039398] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12248.041141] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12248.061891] Lustre: Skipped 1 previous similar message [12248.097517] Lustre: Skipped 1 previous similar message [12252.876650] Lustre: server umount lustre-OST0000 complete [12258.273687] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12258.286136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12258.304795] Lustre: Skipped 1 previous similar message [12262.943565] Lustre: server umount lustre-MDT0000 complete [12267.103790] LustreError: 354015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790008078 with bad export cookie 14584825324404234421 [12267.109334] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12267.124086] LustreError: 354015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12267.436487] Lustre: server umount lustre-MDT0001 complete [12276.684376] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 12:28:06 (1790008086) [12294.224755] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [12307.710164] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [12345.346718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12345.613986] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12345.659539] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12345.785824] Lustre: lustre-MDT0000: new disk, initializing [12345.856244] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12345.883191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12350.883934] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12361.924443] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12362.066648] Lustre: 358764:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12362.112931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12362.116785] Lustre: Skipped 1 previous similar message [12362.169690] Lustre: lustre-MDT0001: new disk, initializing [12362.231317] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12362.276144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12362.293305] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12367.867809] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12377.982107] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12384.166589] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12391.282383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12391.550578] Lustre: lustre-OST0000: new disk, initializing [12391.557614] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12391.566711] Lustre: 360733:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12391.643982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12396.590777] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [12399.683390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12399.693417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12399.836838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12406.171044] Lustre: DEBUG MARKER: oleg344-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12446.194222] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12446.199562] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12446.207217] LustreError: Skipped 1 previous similar message [12446.208315] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12446.214550] Lustre: Skipped 1 previous similar message [12446.231546] Lustre: Skipped 2 previous similar messages [12449.047454] Lustre: server umount lustre-OST0000 complete [12454.369809] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12454.384670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12454.408947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12459.638950] Lustre: server umount lustre-MDT0000 complete [12461.544091] LustreError: 361602:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12461.571848] LustreError: 361602:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12463.284829] LustreError: 358757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790008274 with bad export cookie 14584825324404235303 [12463.294596] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12463.299329] LustreError: 358757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12463.551676] Lustre: server umount lustre-MDT0001 complete [12475.280995] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12478.342928] Key type lgssc unregistered [12478.682202] LNet: 362909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12478.686819] LNetError: 362909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12478.700888] LNet: Removed LNI 192.168.203.144@tcp [12479.596217] Key type .llcrypt unregistered [12479.600103] Key type ._llcrypt unregistered [12571.036311] Key type ._llcrypt registered [12571.044089] Key type .llcrypt registered [12571.158739] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12571.410367] Key type .llcrypt unregistered [12571.418951] Key type ._llcrypt unregistered [12650.417181] Key type ._llcrypt registered [12650.419967] Key type .llcrypt registered [12650.568185] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12650.870428] Key type .llcrypt unregistered [12650.874842] Key type ._llcrypt unregistered [12682.181892] Key type ._llcrypt registered [12682.185639] Key type .llcrypt registered [12682.297281] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12682.606323] Key type .llcrypt unregistered [12682.611177] Key type ._llcrypt unregistered [12712.240540] Key type ._llcrypt registered [12712.244905] Key type .llcrypt registered [12712.418151] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12712.787106] Key type .llcrypt unregistered [12712.789099] Key type ._llcrypt unregistered [12788.977672] Key type ._llcrypt registered [12788.982752] Key type .llcrypt registered [12789.155458] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12789.534477] Key type .llcrypt unregistered [12789.538056] Key type ._llcrypt unregistered [12818.770079] Key type ._llcrypt registered [12818.772094] Key type .llcrypt registered [12818.883044] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12819.199326] Key type .llcrypt unregistered [12819.202535] Key type ._llcrypt unregistered [12850.781534] Key type ._llcrypt registered [12850.786618] Key type .llcrypt registered [12850.891658] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12851.190432] Key type .llcrypt unregistered [12851.195797] Key type ._llcrypt unregistered [12880.997937] Key type ._llcrypt registered [12881.002222] Key type .llcrypt registered [12881.095221] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12881.385600] Key type .llcrypt unregistered [12881.387595] Key type ._llcrypt unregistered [12909.251618] Key type ._llcrypt registered [12909.253222] Key type .llcrypt registered [12909.372725] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12909.672406] Key type .llcrypt unregistered [12909.677532] Key type ._llcrypt unregistered [12939.177814] Key type ._llcrypt registered [12939.181357] Key type .llcrypt registered [12939.322679] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [12939.582376] Key type .llcrypt unregistered [12939.588151] Key type ._llcrypt unregistered [13005.883537] Key type ._llcrypt registered [13005.885388] Key type .llcrypt registered [13005.981176] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13006.363290] Key type .llcrypt unregistered [13006.368586] Key type ._llcrypt unregistered [13041.054445] Key type ._llcrypt registered [13041.056784] Key type .llcrypt registered [13041.159640] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13041.495468] Key type .llcrypt unregistered [13041.499818] Key type ._llcrypt unregistered [13073.738627] Key type ._llcrypt registered [13073.743372] Key type .llcrypt registered [13073.869537] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13074.283338] Key type .llcrypt unregistered [13074.285466] Key type ._llcrypt unregistered [13127.202323] Key type ._llcrypt registered [13127.209234] Key type .llcrypt registered [13127.320897] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13141.141046] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13141.948461] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13142.048326] alg: No test for adler32 (adler32-zlib) [13143.154403] Lustre: Lustre: Build Version: 2.17.58_63_g3df9749 [13143.389248] LNet: Added LNI 192.168.203.144@tcp [8/256/0/180] [13145.064204] Key type lgssc registered [13146.087077] Lustre: Echo OBD driver; http://www.lustre.org/ [13181.867814] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13181.900125] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13183.182546] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [13183.213854] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [13183.304163] Lustre: tst250fs-MDT0000: new disk, initializing [13183.374445] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13183.391782] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13187.995665] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13198.526756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13198.651277] Lustre: 383168:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13198.738746] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [13198.743333] Lustre: Skipped 1 previous similar message [13198.851722] Lustre: tst250fs-MDT0001: new disk, initializing [13198.984495] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13199.071844] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13199.085192] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13203.177461] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13208.521305] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [13217.702061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13218.277960] Lustre: tst250fs-OST0000: new disk, initializing [13218.282956] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [13218.290190] Lustre: 384800:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13218.422147] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13224.038155] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13224.060357] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13224.180800] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13226.115523] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13238.638581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13238.871195] Lustre: tst250fs-OST0001: new disk, initializing [13238.874228] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [13238.884533] Lustre: 385673:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13239.047705] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13246.664305] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_default_debug -1 all [13247.582993] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13247.598422] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13247.761790] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13252.915358] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [13255.761674] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [13255.769799] Lustre: Skipped 2 previous similar messages [13275.621290] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13275.632101] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13275.645658] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13278.698835] Lustre: tst250fs-MDT0000-lwp-OST0000: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13278.704364] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [13278.716260] Lustre: Skipped 2 previous similar messages [13278.737091] Lustre: Skipped 3 previous similar messages [13281.480274] Lustre: server umount tst250fs-MDT0000 complete [13285.009368] LustreError: 383162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790009096 with bad export cookie 3059870187607759834 [13285.020810] LustreError: 383162:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13285.021541] LustreError: MGC192.168.203.144@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13285.434992] Lustre: server umount tst250fs-MDT0001 complete [13299.578043] Lustre: server umount tst250fs-OST0000 complete [13313.836937] Lustre: server umount tst250fs-OST0001 complete [13333.040492] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13341.145584] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13372.235336] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 12:46:22 (1790009182) [13373.699837] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [13389.721569] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing set_hostid [13399.071436] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing load_modules_local [13432.582281] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13175 sec ========= 12:47:22 (1790009242) [13434.373268] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 12:47:24 (1790009244) === [13438.569619] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 12:47:28 (1790009248) === [13476.330286] Lustre: DEBUG MARKER: oleg344-server.virtnet: executing unload_modules_local [13479.201628] Key type lgssc unregistered [13479.549698] LNet: 394814:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13479.559339] LNetError: 394814:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13479.588677] LNet: Removed LNI 192.168.203.144@tcp [13480.354486] Key type .llcrypt unregistered [13480.359967] Key type ._llcrypt unregistered