[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 1720302678 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002000] x2apic enabled [ 0.002000] Switched APIC routing to physical x2apic. [ 0.002000] kvm-guest: setup PV IPIs [ 0.002000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.002000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.002000] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.002000] pid_max: default: 32768 minimum: 301 [ 0.002000] LSM: Security Framework initializing [ 0.002000] Yama: becoming mindful. [ 0.002000] SELinux: Initializing. [ 0.002000] *** VALIDATE selinux *** [ 0.002000] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.002000] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.002000] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.002000] *** VALIDATE tmpfs *** [ 0.002000] *** VALIDATE proc *** [ 0.002000] *** VALIDATE cgroup *** [ 0.002000] *** VALIDATE cgroup2 *** [ 0.002000] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.002000] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.002000] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.002000] Spectre V2 : User space: Vulnerable [ 0.002000] Speculative Store Bypass: Vulnerable [ 0.002000] debug: unmapping init [mem 0xffffffffae259000-0xffffffffae260fff] [ 0.002000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.002599] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.003021] ... version: 2 [ 0.004010] ... bit width: 48 [ 0.005008] ... generic registers: 4 [ 0.006008] ... value mask: 0000ffffffffffff [ 0.007020] ... max period: 00007fffffffffff [ 0.008012] ... fixed-purpose events: 3 [ 0.009009] ... event mask: 000000070000000f [ 0.010270] rcu: Hierarchical SRCU implementation. [ 0.012382] smp: Bringing up secondary CPUs ... [ 0.013500] x86: Booting SMP configuration: [ 0.014017] .... node #0, CPUs: #1 #2 #3 [ 0.023260] smp: Brought up 1 node, 4 CPUs [ 0.025011] smpboot: Max logical packages: 1 [ 0.026014] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.069014] node 0 deferred pages initialised in 41ms [ 0.071265] devtmpfs: initialized [ 0.072259] x86/mm: Memory block size: 128MB [ 0.074774] gcov: version magic: 0x41383552 [ 0.076430] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.077064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.078211] pinctrl core: initialized pinctrl subsystem [ 0.079158] [ 0.080009] ************************************************************* [ 0.081009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.082009] ** ** [ 0.083009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.084011] ** ** [ 0.085016] ** This means that this kernel is built to expose internal ** [ 0.086012] ** IOMMU data structures, which may compromise security on ** [ 0.087009] ** your system. ** [ 0.088013] ** ** [ 0.089013] ** If you see this message and you are not debugging the ** [ 0.090015] ** kernel, report this immediately to your vendor! ** [ 0.091013] ** ** [ 0.092012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.093013] ************************************************************* [ 0.094766] NET: Registered protocol family 16 [ 0.095453] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.096059] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.097053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.098672] cpuidle: using governor menu [ 0.101043] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.102445] PCI: Using configuration type 1 for base access [ 0.103123] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.110126] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.112024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.117006] cryptd: max_cpu_qlen set to 1000 [ 0.120015] ACPI: Added _OSI(Module Device) [ 0.121011] ACPI: Added _OSI(Processor Device) [ 0.122012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.123010] ACPI: Added _OSI(Processor Aggregator Device) [ 0.127574] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.131408] ACPI: Interpreter enabled [ 0.132073] ACPI: PM: (supports S0 S3 S4 S5) [ 0.133014] ACPI: Using IOAPIC for interrupt routing [ 0.134103] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.135430] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.146136] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.147041] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.148016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.149080] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.151385] acpiphp: Slot [2] registered [ 0.152143] acpiphp: Slot [5] registered [ 0.153138] acpiphp: Slot [6] registered [ 0.154101] acpiphp: Slot [7] registered [ 0.155145] acpiphp: Slot [8] registered [ 0.156108] acpiphp: Slot [9] registered [ 0.157119] acpiphp: Slot [10] registered [ 0.158124] acpiphp: Slot [3] registered [ 0.159107] acpiphp: Slot [4] registered [ 0.160114] acpiphp: Slot [11] registered [ 0.161132] acpiphp: Slot [12] registered [ 0.162112] acpiphp: Slot [13] registered [ 0.163114] acpiphp: Slot [14] registered [ 0.164138] acpiphp: Slot [15] registered [ 0.165125] acpiphp: Slot [16] registered [ 0.166102] acpiphp: Slot [17] registered [ 0.167133] acpiphp: Slot [18] registered [ 0.168113] acpiphp: Slot [19] registered [ 0.169097] acpiphp: Slot [20] registered [ 0.170096] acpiphp: Slot [21] registered [ 0.171118] acpiphp: Slot [22] registered [ 0.172144] acpiphp: Slot [23] registered [ 0.173119] acpiphp: Slot [24] registered [ 0.174274] acpiphp: Slot [25] registered [ 0.175243] acpiphp: Slot [26] registered [ 0.176114] acpiphp: Slot [27] registered [ 0.177129] acpiphp: Slot [28] registered [ 0.178123] acpiphp: Slot [29] registered [ 0.179105] acpiphp: Slot [30] registered [ 0.180126] acpiphp: Slot [31] registered [ 0.181072] PCI host bridge to bus 0000:00 [ 0.182019] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.183027] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.184021] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.185024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.186026] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.187027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.188185] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.190246] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.192549] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.198014] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.201051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.202019] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.203024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.204022] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.205550] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.206866] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.207047] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.208714] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.211012] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.220013] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.222014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.226114] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.234014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.304019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.335050] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.357009] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.377014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.396014] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.466012] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.493171] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.509013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.522012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.572015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.596048] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.613014] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.640015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.678011] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.697000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.707015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.713022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.736016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.745835] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.752013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.757011] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.770996] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.779151] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.781321] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.783286] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.785250] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.786142] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.794003] iommu: Default domain type: Passthrough [ 0.796195] SCSI subsystem initialized [ 0.797000] ACPI: bus type USB registered [ 0.799164] usbcore: registered new interface driver usbfs [ 0.801083] usbcore: registered new interface driver hub [ 0.804097] usbcore: registered new device driver usb [ 0.806189] pps_core: LinuxPPS API ver. 1 registered [ 0.809038] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.812151] PTP clock support registered [ 0.814304] EDAC MC: Ver: 3.0.0 [ 0.816323] PCI: Using ACPI for IRQ routing [ 0.819009] NetLabel: Initializing [ 0.821015] NetLabel: domain hash size = 128 [ 0.823015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.826139] NetLabel: unlabeled traffic allowed by default [ 0.828336] vgaarb: loaded [ 0.831123] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.833107] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.841139] clocksource: Switched to clocksource kvm-clock [ 0.961327] VFS: Disk quotas dquot_6.6.0 [ 0.962799] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.964941] *** VALIDATE ramfs *** [ 0.966094] *** VALIDATE hugetlbfs *** [ 0.967503] pnp: PnP ACPI init [ 0.969649] pnp: PnP ACPI: found 6 devices [ 0.984250] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.987052] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.988980] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.990931] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.992893] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.995088] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.997530] NET: Registered protocol family 2 [ 0.999623] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.004377] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.008073] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.013548] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.016751] TCP: Hash tables configured (established 65536 bind 65536) [ 1.019571] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.022570] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.025251] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.027885] NET: Registered protocol family 1 [ 1.030677] RPC: Registered named UNIX socket transport module. [ 1.032363] RPC: Registered udp transport module. [ 1.034168] RPC: Registered tcp transport module. [ 1.035820] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.037871] NET: Registered protocol family 44 [ 1.039590] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.041948] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.044678] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.049218] PCI: CLS 0 bytes, default 64 [ 1.051357] Unpacking initramfs... [ 4.770800] debug: unmapping init [mem 0xffff981fbcc54000-0xffff981fbffbffff] [ 4.785932] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 4.795657] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 4.804607] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 6.742873] Initialise system trusted keyrings [ 6.749380] Key type blacklist registered [ 6.760983] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 6.809615] zbud: loaded [ 6.858933] *** VALIDATE nfs *** [ 6.861290] *** VALIDATE nfs4 *** [ 6.864228] pstore: using deflate compression [ 6.870867] Platform Keyring initialized [ 7.362628] NET: Registered protocol family 38 [ 7.371372] Key type asymmetric registered [ 7.378585] Asymmetric key parser 'x509' registered [ 7.387222] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 7.396965] io scheduler mq-deadline registered [ 7.408236] io scheduler kyber registered [ 7.415693] io scheduler bfq registered [ 7.422360] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 7.432417] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 7.443512] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 7.453202] ACPI: Power Button [PWRF] [ 7.472575] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 7.490986] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 7.680533] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 7.700054] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 7.799779] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 7.837205] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 7.871276] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 7.924585] Non-volatile memory driver v1.3 [ 7.927706] Linux agpgart interface v0.103 [ 7.969878] virtio_blk virtio1: [vda] 150000 512-byte logical blocks (76.8 MB/73.2 MiB) [ 7.972874] vda: detected capacity change from 0 to 76800000 [ 7.995827] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 7.998514] vdb: detected capacity change from 0 to 1073741824 [ 8.019631] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.023912] vdc: detected capacity change from 0 to 2621440000 [ 8.043216] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 8.046512] vdd: detected capacity change from 0 to 2621440000 [ 8.062996] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.067075] vde: detected capacity change from 0 to 4294967296 [ 8.084643] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 8.087111] vdf: detected capacity change from 0 to 4294967296 [ 8.093479] libphy: Fixed MDIO Bus: probed [ 8.149505] usbcore: registered new interface driver usbserial_generic [ 8.152478] usbserial: USB Serial support registered for generic [ 8.154747] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 8.162503] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 8.164499] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 8.168900] mousedev: PS/2 mouse device common for all mice [ 8.171854] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 8.173486] rtc_cmos 00:05: RTC can wake from S4 [ 8.192496] rtc_cmos 00:05: registered as rtc0 [ 8.198436] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 8.199163] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 8.212747] intel_pstate: CPU model not supported [ 8.217656] hid: raw HID events driver (C) Jiri Kosina [ 8.220876] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 8.223466] usbcore: registered new interface driver usbhid [ 8.229461] usbhid: USB HID core driver [ 8.232724] drop_monitor: Initializing network drop monitor service [ 8.234792] Initializing XFRM netlink socket [ 8.238761] NET: Registered protocol family 10 [ 8.247240] Segment Routing with IPv6 [ 8.250281] NET: Registered protocol family 17 [ 8.256156] mpls_gso: MPLS GSO support [ 8.265909] RAS: Correctable Errors collector initialized. [ 8.270445] AVX version of gcm_enc/dec engaged. [ 8.272812] AES CTR mode by8 optimization enabled [ 8.362851] sched_clock: Marking stable (8362832589, 0)->(10786555014, -2423722425) [ 8.367568] registered taskstats version 1 [ 8.370311] Loading compiled-in X.509 certificates [ 8.372598] zswap: loaded using pool lzo/zbud [ 8.403725] Key type big_key registered [ 8.419484] Key type encrypted registered [ 8.421902] ima: No TPM chip found, activating TPM-bypass! [ 8.424494] ima: Allocated hash algorithm: sha1 [ 8.426540] ima: No architecture policies found [ 8.429223] evm: Initialising EVM extended attributes: [ 8.431551] evm: security.selinux [ 8.433386] evm: security.ima [ 8.434908] evm: security.capability [ 8.437116] evm: HMAC attrs: 0x1 [ 8.440763] rtc_cmos 00:05: setting system clock to 2026-09-17 08:49:57 UTC (1789634997) [ 8.451125] debug: unmapping init [mem 0xffffffffaf203000-0xffffffffaf3fffff] [ 8.455613] debug: unmapping init [mem 0xffffffffadf82000-0xffffffffae258fff] [ 8.460148] Write protecting the kernel read-only data: 28672k [ 8.463622] debug: unmapping init [mem 0xffffffffac603000-0xffffffffac7fffff] [ 8.466811] debug: unmapping init [mem 0xffffffffacf14000-0xffffffffacffffff] [ 8.510814] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 8.523402] systemd[1]: Detected virtualization kvm. [ 8.525317] systemd[1]: Detected architecture x86-64. [ 8.528350] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 8.562392] systemd[1]: No hostname configured. [ 8.564462] systemd[1]: Set hostname to . [ 8.566747] random: systemd: uninitialized urandom read (16 bytes read) [ 8.569306] systemd[1]: Initializing machine ID from random generator. [ 8.691077] random: ln: uninitialized urandom read (6 bytes read) [ 8.898773] random: systemd: uninitialized urandom read (16 bytes read) [ 8.903243] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 8.907993] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 8.912222] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Apply Kernel Variables... Starting Setup Virtual Console... Starting Journal Service... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 11.003434] device-mapper: uevent: version 1.0.3 [ 11.006224] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 13.281718] virtio_net virtio0 ens2: renamed from eth0 [ 13.342326] random: fast init done [ 14.198601] scsi host0: ata_piix [ 14.638808] scsi host1: ata_piix [ 14.640225] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 14.655679] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 18.956564] random: crng init done [ 18.960260] random: 7 urandom warning(s) missed due to ratelimiting [ 22.266562] dracut-initqueue[592]: RTNETLINK answers: File exists [ 22.657013] hrtimer: interrupt took 11003193 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 24.362068] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 27.494199] printk: systemd: 26 output lines suppressed due to ratelimiting [ 28.292434] SELinux: Disabled at runtime. [ 28.520256] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 28.544601] systemd[1]: Detected virtualization kvm. [ 28.554301] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 30.828384] systemd[1]: initrd-switch-root.service: Succeeded. [ 30.833267] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 30.847760] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 30.854359] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 30.864321] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 30.871521] systemd[1]: Starting Journal Service... Starting Journal Service... [ 30.886504] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Created slice system-getty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Mounting POSIX Message Queue File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Reached target Slices. Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ 31.406165] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... Starting Create Static Device Nodes in /dev... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 32.499324] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 34.056020] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 34.221054] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 35.758711] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 35.925257] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (8s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (9s / no limit)[ 40.163228] Key type dns_resolver registered [ *** ] A start job is running for Configur…only root support (10s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 41.587484] NFS: Registering the id_resolver key type [ 41.596668] Key type id_resolver registered [ 41.598820] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (11s / no limit) [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg339-server login: [ 106.747628] libcfs: loading out-of-tree module taints kernel. [ 106.813205] Key type ._llcrypt registered [ 106.815815] Key type .llcrypt registered [ 106.917176] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 122.842822] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 124.101474] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 124.112654] alg: No test for adler32 (adler32-zlib) [ 125.443799] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 126.290996] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 128.007842] Key type lgssc registered [ 129.607357] Lustre: Echo OBD driver; http://www.lustre.org/ [ 145.444473] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 182.435346] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 194.417218] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 194.442194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 195.745131] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 195.805238] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 195.872310] Lustre: lustre-MDT0000: new disk, initializing [ 195.977259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 195.995775] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 200.495513] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 212.469130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 212.550882] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 212.590736] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 212.595947] Lustre: Skipped 1 previous similar message [ 212.675765] Lustre: lustre-MDT0001: new disk, initializing [ 212.741660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 212.776314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 212.785374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 216.741242] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 221.245954] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 229.466549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 229.644428] Lustre: lustre-OST0000: new disk, initializing [ 229.647058] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 229.651930] Lustre: 8452:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 229.696744] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 235.680468] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 236.035098] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 236.043465] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 236.085377] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 249.514791] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 249.655231] Lustre: lustre-OST0001: new disk, initializing [ 249.660814] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 249.668993] Lustre: 9524:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 249.735916] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 255.224841] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.663905] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 257.683180] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 257.793066] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 273.936174] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 284.329228] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 293.726968] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing check_logdir /tmp/testlogs/ [ 301.459218] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing yml_node [ 307.647859] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 310.697147] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 314.122493] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 315.995694] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Sep 17 04:55:04 EDT 2026 [ 331.787354] Lustre: DEBUG MARKER: excepting tests: 32newtarball [ 334.026981] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 355.297970] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 355.302543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 355.311241] Lustre: Skipped 1 previous similar message [ 359.059531] Lustre: server umount lustre-MDT0000 complete [ 362.946844] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635352 with bad export cookie 16733756202242160664 [ 362.951239] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 362.954854] LustreError: 6509:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 365.539709] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 365.546510] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 365.546980] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 365.546986] Lustre: Skipped 3 previous similar messages [ 365.561151] Lustre: Skipped 3 previous similar messages [ 365.605363] LustreError: 6524:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 367.584706] LustreError: 10103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 367.585054] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 367.597893] LustreError: 10103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 367.605837] Lustre: Skipped 2 previous similar messages [ 372.705339] LustreError: 8829:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.706189] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.713692] LustreError: 8829:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 372.723421] Lustre: Skipped 1 previous similar message [ 377.311205] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 377.567412] Lustre: server umount lustre-MDT0001 complete [ 381.174039] Lustre: server umount lustre-OST0000 complete [ 384.566655] Lustre: server umount lustre-OST0001 complete [ 392.818368] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 408.029794] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 449.116496] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 449.377649] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 449.403816] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 449.466336] Lustre: lustre-MDT0000: new disk, initializing [ 449.533180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 449.553122] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 454.049589] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 464.351208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 464.538978] Lustre: 15103:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 464.621086] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 464.628084] Lustre: Skipped 1 previous similar message [ 464.721735] Lustre: lustre-MDT0001: new disk, initializing [ 464.821094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 464.870535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 464.894461] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 469.685650] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 478.977794] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 484.273631] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 490.119157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 490.413797] Lustre: lustre-OST0000: new disk, initializing [ 490.416939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 490.423922] Lustre: 17076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 490.495973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 492.097943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 492.112680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 492.306286] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 497.992257] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 507.121719] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 512.136574] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 512.414065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 517.909432] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 518.198723] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 523.233438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 523.244112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 523.246947] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 523.256333] Lustre: Skipped 1 previous similar message [ 523.260296] Lustre: Skipped 2 previous similar messages [ 526.207348] Lustre: server umount lustre-OST0000 complete [ 531.424966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 531.436893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 531.451689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 531.461600] Lustre: Skipped 1 previous similar message [ 535.634849] Lustre: server umount lustre-MDT0000 complete [ 538.594079] LustreError: 15111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 538.607037] LustreError: 15111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 538.653891] LustreError: 17074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635527 with bad export cookie 16733756202242162197 [ 538.657836] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 538.675736] LustreError: 17074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 539.033766] Lustre: server umount lustre-MDT0001 complete [ 544.192715] Lustre: DEBUG MARKER: == conf-sanity test 121: failover MGS ==================== 04:58:52 (1789635532) [ 561.243925] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 561.568960] LustreError: 19948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 561.617097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 565.764461] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 571.873455] LustreError: 19949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 571.900590] LustreError: 19949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 573.017028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 577.659047] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 584.177365] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 589.802961] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 591.991985] Lustre: Failing over lustre-MDT0000 [ 592.309099] Lustre: server umount lustre-MDT0000 complete [ 593.889249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 593.898457] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 593.910467] Lustre: Skipped 1 previous similar message [ 593.913646] LustreError: 19949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.927190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 612.118898] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 612.557855] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 617.358924] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 617.954629] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 617.958732] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 618.009791] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 627.072689] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 632.303862] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 632.308166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 632.322222] Lustre: Skipped 2 previous similar messages [ 632.333202] Lustre: Skipped 3 previous similar messages [ 637.362472] Lustre: server umount lustre-MDT0000 complete [ 637.409260] LustreError: 19949:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 637.429480] LustreError: 19949:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 641.172281] LustreError: 19930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635630 with bad export cookie 16733756202242163611 [ 641.175092] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 641.195834] LustreError: 19930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 641.764146] Lustre: server umount lustre-MDT0001 complete [ 650.347509] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 651.048443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 651.056522] Lustre: Skipped 2 previous similar messages [ 655.325338] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 663.564310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 668.706209] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 677.484101] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 684.246834] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 686.180788] Lustre: Failing over lustre-MDT0000 [ 686.404639] Lustre: server umount lustre-MDT0000 complete [ 686.560400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 703.968081] Lustre: 3651:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789635677/real 1789635677] req@ffff982003064700 x1876568430542464/t0(0) o400->MGC192.168.203.139@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1789635693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 703.969648] LustreError: 23282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.988792] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 704.006753] LustreError: 23282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 704.976735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 715.550202] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 719.369199] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 720.870742] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 720.879800] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 720.898134] Lustre: Skipped 1 previous similar message [ 720.931515] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. [ 728.147837] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mgc.*.mgs_server_uuid 1475 0 [ 733.926780] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 733.947222] Lustre: Skipped 1 previous similar message [ 733.954820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 733.964767] Lustre: Skipped 1 previous similar message [ 738.301979] Lustre: server umount lustre-MDT0000 complete [ 742.349198] LustreError: 23262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635731 with bad export cookie 16733756202242164549 [ 742.362706] LustreError: 23262:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 742.363631] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 742.788407] Lustre: server umount lustre-MDT0001 complete [ 751.182604] Lustre: DEBUG MARKER: == conf-sanity test 122a: Check OST sequence update ====== 05:02:20 (1789635740) [ 765.680354] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 775.953915] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 813.086262] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 823.344892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 823.532472] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 823.566670] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 823.617387] Lustre: lustre-MDT0000: new disk, initializing [ 823.705243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 823.710707] Lustre: Skipped 2 previous similar messages [ 823.727950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 827.270537] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 837.003809] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 837.107091] Lustre: 29675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 837.458252] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 842.080824] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 851.565966] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 857.100833] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 863.173049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 863.422138] Lustre: 31644:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 864.480810] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 864.520887] Lustre: *** cfs_fail_loc=1e0, val=0*** [ 864.558834] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 864.573135] Lustre: lustre-OST0000: MDS LAST_ID [0x280000401:0x21:0x0] (33) is 33554143 behind OST LAST_ID [0x280000401:0x1ffff00:0x0] (33554176), trust the OST [ 868.087915] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 875.502976] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 880.928250] Lustre: lustre-OST0000-osc-MDT0001: update sequence from 0x280000400 to 0x280000402 [ 900.577802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.594469] Lustre: Skipped 1 previous similar message [ 900.600643] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 900.601175] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.613570] Lustre: Skipped 1 previous similar message [ 906.599329] Lustre: server umount lustre-OST0000 complete [ 915.835419] Lustre: server umount lustre-MDT0000 complete [ 915.945627] LustreError: 29682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.977325] LustreError: 29682:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 919.368880] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789635908 with bad export cookie 16733756202242164948 [ 919.373508] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 919.376421] LustreError: 29670:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 919.733424] Lustre: server umount lustre-MDT0001 complete [ 929.988264] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 932.626181] Key type lgssc unregistered [ 932.886790] LNet: 33618:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 932.905715] LNetError: 33618:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 932.917353] LNet: Removed LNI 192.168.203.139@tcp [ 933.858144] Key type .llcrypt unregistered [ 933.860131] Key type ._llcrypt unregistered [ 955.771855] Key type ._llcrypt registered [ 955.776532] Key type .llcrypt registered [ 955.870507] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 956.718685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 956.844487] alg: No test for adler32 (adler32-zlib) [ 957.888783] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 958.085814] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 959.727223] Key type lgssc registered [ 960.607903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 967.222251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 967.240336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 968.634330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 972.833810] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 978.920743] LustreError: 35018:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 978.940315] LustreError: 35018:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 980.385237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 980.681675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 984.083163] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 990.030404] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 994.687618] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1000.087127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1000.359359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1005.421315] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1005.510376] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000402:938 to 0x280000402:1025) [ 1011.637697] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1013.727913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000403 [ 1027.960604] Lustre: DEBUG MARKER: == conf-sanity test 123ab: llog_print params output values from set_param -P ========================================================== 05:06:57 (1789636017) [ 1029.499100] Lustre: Modifying parameter general.jobid_name=TESTNAME in log params [ 1030.825627] Lustre: Modifying parameter general.jobid_name=jobid_name=%e.root in log params [ 1035.598451] Lustre: DEBUG MARKER: == conf-sanity test 123ac: llog_print with --start and --end ========================================================== 05:07:04 (1789636024) [ 1042.163984] Lustre: DEBUG MARKER: == conf-sanity test 123ad: llog_print shows all records == 05:07:11 (1789636031) [ 1042.929426] Lustre: Setting parameter lustre-OST0000-osc.osc.max_dirty_mb=462 in log lustre-client [ 1049.725235] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_dirty_mb= in log lustre-client [ 1049.730116] Lustre: Skipped 2 previous similar messages [ 1051.155409] Lustre: DEBUG MARKER: == conf-sanity test 123ae: llog_cancel can cancel requested record ========================================================== 05:07:20 (1789636040) [ 1054.129501] Lustre: Setting parameter general.osc.*.max_dirty_mb=462 in log params [ 1054.133963] Lustre: Skipped 3 previous similar messages [ 1069.832122] Lustre: Disabling parameter lustre-OST0000-osc.osc.max_pages_per_rpc= in log lustre-client [ 1069.836920] Lustre: Skipped 6 previous similar messages [ 1071.802305] Lustre: DEBUG MARKER: == conf-sanity test 123af: llog_catlist can show all config files correctly ========================================================== 05:07:41 (1789636061) [ 1074.133330] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1076.910734] Lustre: *** cfs_fail_loc=131b, val=2*** [ 1083.792958] Lustre: DEBUG MARKER: == conf-sanity test 123ag: llog_print skips values deleted by set_param -P -d ========================================================== 05:07:52 (1789636072) [ 1086.360855] Lustre: Disabling parameter general.jobid_name= in log params [ 1086.370712] Lustre: Skipped 4 previous similar messages [ 1097.958792] Lustre: DEBUG MARKER: == conf-sanity test 123ah: del_ost cancels config log entries correctly ========================================================== 05:08:06 (1789636086) [ 1111.010221] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.012386] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1111.032264] Lustre: Skipped 1 previous similar message [ 1111.049987] Lustre: Skipped 1 previous similar message [ 1115.890813] Lustre: server umount lustre-OST0000 complete [ 1121.254958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.265891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.267360] Lustre: Skipped 1 previous similar message [ 1125.731248] Lustre: server umount lustre-MDT0000 complete [ 1126.368867] LustreError: 35019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.389067] LustreError: 35019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1128.872089] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636117 with bad export cookie 1609705195874608280 [ 1128.877455] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1128.885616] LustreError: 35000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1129.249745] Lustre: server umount lustre-MDT0001 complete [ 1137.859197] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1140.341492] Key type lgssc unregistered [ 1140.610391] LNet: 42877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1140.618285] LNetError: 42877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1140.639672] LNet: Removed LNI 192.168.203.139@tcp [ 1141.387182] Key type .llcrypt unregistered [ 1141.391130] Key type ._llcrypt unregistered [ 1155.065782] Key type ._llcrypt registered [ 1155.068102] Key type .llcrypt registered [ 1155.146295] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 1167.149421] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1167.816966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1167.834478] alg: No test for adler32 (adler32-zlib) [ 1168.786487] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1168.970629] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1170.623631] Key type lgssc registered [ 1171.354757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1197.270617] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1197.290251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1198.470358] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1198.499318] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1198.546574] Lustre: lustre-MDT0000: new disk, initializing [ 1198.619967] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1198.630859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1202.094556] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1210.993091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1211.082508] Lustre: 46004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1211.142870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1211.151471] Lustre: Skipped 1 previous similar message [ 1211.224904] Lustre: lustre-MDT0001: new disk, initializing [ 1211.343269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1211.391516] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1211.399444] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1215.327313] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1222.453595] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1226.628454] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1231.989921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1232.220586] Lustre: lustre-OST0000: new disk, initializing [ 1232.226735] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1232.245949] Lustre: 47969:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1232.342565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1237.057853] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1239.607745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 1239.622832] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 1239.753742] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 1245.119497] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1249.107871] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 1249.260911] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 1252.786831] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 1252.954940] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 1255.399945] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.401688] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1255.435062] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1255.447991] Lustre: Skipped 1 previous similar message [ 1260.519559] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1260.530359] Lustre: Skipped 1 previous similar message [ 1260.721616] Lustre: server umount lustre-OST0000 complete [ 1265.633817] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1265.643880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1265.651849] Lustre: Skipped 2 previous similar messages [ 1265.664057] Lustre: Skipped 1 previous similar message [ 1270.719064] Lustre: server umount lustre-MDT0000 complete [ 1273.642068] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636262 with bad export cookie 12197758520187141458 [ 1273.646210] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.656467] LustreError: 45994:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1273.890394] Lustre: server umount lustre-MDT0001 complete [ 1281.642333] Lustre: DEBUG MARKER: == conf-sanity test 123ai: llog_print display all non skipped records ========================================================== 05:11:10 (1789636270) [ 1288.066550] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1288.494875] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1288.514200] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1288.586802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.957181] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1293.801822] LustreError: 50429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1298.919760] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1300.323400] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1300.601887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1304.243761] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1310.422724] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1315.326526] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1320.529251] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1320.827629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1325.726502] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1331.547664] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1340.428504] LustreError: 52956:0:(llog.c:449:llog_init_handle()) llog type is not specified! [ 1341.383572] Lustre: Modifying parameter general.timeout=1 in log params [ 1343.029491] Lustre: Modifying parameter general.timeout=3 in log params [ 1343.034495] Lustre: Skipped 1 previous similar message [ 1345.466691] Lustre: Modifying parameter general.timeout=6 in log params [ 1345.477223] Lustre: Skipped 2 previous similar messages [ 1349.635712] Lustre: Modifying parameter general.timeout=11 in log params [ 1349.638701] Lustre: Skipped 4 previous similar messages [ 1357.970636] Lustre: Modifying parameter general.timeout=22 in log params [ 1357.973808] Lustre: Skipped 10 previous similar messages [ 1374.568736] Lustre: Modifying parameter general.timeout=45 in log params [ 1374.579070] Lustre: Skipped 22 previous similar messages [ 1406.956212] Lustre: Modifying parameter general.timeout=95 in log params [ 1406.961230] Lustre: Skipped 49 previous similar messages [ 1437.454876] Lustre: DEBUG MARKER: == conf-sanity test 123aj: check permanent TBF rules ===== 05:13:46 (1789636426) [ 1439.648257] Lustre: 59886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule4 jobid={fio.0.0} rate=500 (mode = 2) failed: rc = -17 [ 1441.072357] Lustre: 59984:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule2 nid={10.10.2.20@tcp} rate=1000 (mode = 2) failed: rc = -17 [ 1443.655925] Lustre: 60181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule=start rule1 uid={500 502 } rate=1000 (mode = 2) failed: rc = -17 [ 1443.675143] Lustre: 60181:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 1 previous similar message [ 1465.908486] Lustre: 61739:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/ost.OSS.ost_io.nrs_tbf_rule= (mode = 1) failed: rc = -2 [ 1465.921448] Lustre: 61739:0:(mgs_llog.c:1450:mgs_modify_param()) Skipped 2 previous similar messages [ 1465.955524] LustreError: 61739:0:(mgs_handler.c:1206:mgs_iocontrol()) MGS: setparam err: rc = -2 [ 1468.693779] Lustre: DEBUG MARKER: == conf-sanity test 123F: clear and reset all parameters using set_param -F ========================================================== 05:14:18 (1789636458) [ 1471.334741] Lustre: Setting parameter general.jobid_var=TESTNAME in log params [ 1471.339407] Lustre: Skipped 56 previous similar messages [ 1478.632349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1478.633839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1478.648920] Lustre: Skipped 2 previous similar messages [ 1478.653049] Lustre: Skipped 1 previous similar message [ 1483.072531] Lustre: server umount lustre-MDT0000 complete [ 1483.746170] LustreError: 51156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1483.762880] LustreError: 51156:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1485.574831] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636474 with bad export cookie 12197758520187142347 [ 1485.579986] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1485.582270] LustreError: 50408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1485.839776] Lustre: server umount lustre-MDT0001 complete [ 1499.067851] Lustre: server umount lustre-OST0000 complete [ 1518.378513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1518.533042] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1518.735614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.663865] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1527.586284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1527.650300] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1527.667735] Lustre: 64266:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1530.991634] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1536.055704] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1540.157332] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1544.859047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1544.991452] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1550.490247] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1556.659348] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1558.001389] Lustre: 66834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=TESTNAME (mode = 0) failed: rc = -17 [ 1567.207512] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1567.211100] LustreError: Skipped 1 previous similar message [ 1567.218843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1567.227988] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1567.232803] Lustre: Skipped 2 previous similar messages [ 1569.198978] Lustre: server umount lustre-OST0000 complete [ 1572.321136] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1572.330598] Lustre: Skipped 2 previous similar messages [ 1577.442313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1577.445249] Lustre: Skipped 3 previous similar messages [ 1586.145082] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1586.328351] Lustre: server umount lustre-MDT0000 complete [ 1587.681428] LustreError: 64273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1587.695322] LustreError: 64273:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1589.095756] LustreError: 64258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636578 with bad export cookie 12197758520187145105 [ 1589.098459] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.105556] LustreError: 64258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1589.333574] Lustre: server umount lustre-MDT0001 complete [ 1596.557903] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1598.755738] Key type lgssc unregistered [ 1598.986673] LNet: 68041:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1598.998394] LNetError: 68041:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1599.012267] LNet: Removed LNI 192.168.203.139@tcp [ 1599.655251] Key type .llcrypt unregistered [ 1599.657077] Key type ._llcrypt unregistered [ 1616.096351] Key type ._llcrypt registered [ 1616.100037] Key type .llcrypt registered [ 1616.187942] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1616.825078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1616.904311] alg: No test for adler32 (adler32-zlib) [ 1617.882054] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1618.071227] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1619.703179] Key type lgssc registered [ 1620.396706] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1626.138300] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1626.165756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1627.558815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1630.754319] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1636.753274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1637.275480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1640.370723] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1645.937358] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1650.051656] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1654.970787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1655.389698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1659.750952] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1664.669748] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1679.664777] Lustre: Modifying parameter general.jobid_var=TESTNAME in log params [ 1688.546197] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1688.553978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1688.564784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.570462] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1689.573091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1689.576787] Lustre: Skipped 1 previous similar message [ 1689.588816] Lustre: Skipped 1 previous similar message [ 1691.583972] Lustre: server umount lustre-MDT0000 complete [ 1693.910807] LustreError: 69408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636682 with bad export cookie 8587153566831716221 [ 1693.915643] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1693.924538] LustreError: 69408:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1694.144430] Lustre: server umount lustre-MDT0001 complete [ 1706.957584] Lustre: server umount lustre-OST0000 complete [ 1725.134371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.308978] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1725.334145] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1725.524473] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1728.438857] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1734.613779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.712766] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1734.741480] Lustre: 74319:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1735.127095] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1737.625324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1742.526051] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1746.204427] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1750.747808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1750.892958] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1751.126327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1754.715580] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1759.772434] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1761.126516] Lustre: 76893:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1762.707964] Lustre: Modifying parameter general.jobid_var=disable in log params [ 1766.371217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1766.371432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1766.392542] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1766.396461] Lustre: Skipped 2 previous similar messages [ 1771.493701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1771.501899] Lustre: Skipped 1 previous similar message [ 1772.394825] Lustre: server umount lustre-OST0000 complete [ 1776.101691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1776.106572] LustreError: Skipped 1 previous similar message [ 1776.111235] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1776.126197] Lustre: Skipped 1 previous similar message [ 1776.129440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1781.646734] Lustre: server umount lustre-MDT0000 complete [ 1781.734846] LustreError: 74331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1781.746849] LustreError: 74331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1784.355385] LustreError: 74312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636773 with bad export cookie 8587153566831717054 [ 1784.363178] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1784.363849] LustreError: 74312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1784.662446] Lustre: server umount lustre-MDT0001 complete [ 1792.760324] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1794.776037] Key type lgssc unregistered [ 1795.018598] LNet: 78102:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1795.026104] LNetError: 78102:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1795.037207] LNet: Removed LNI 192.168.203.139@tcp [ 1795.652835] Key type .llcrypt unregistered [ 1795.655165] Key type ._llcrypt unregistered [ 1812.131782] Key type ._llcrypt registered [ 1812.133840] Key type .llcrypt registered [ 1812.212639] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 1812.736564] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1812.782182] alg: No test for adler32 (adler32-zlib) [ 1813.766416] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1813.931589] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1815.559792] Key type lgssc registered [ 1816.323839] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1821.277158] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1821.300707] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1822.721771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1825.683605] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1831.269440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1831.632479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1834.358898] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1839.694654] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1843.385579] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1847.830956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1848.186621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.128403] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 1856.495764] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1876.078380] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1876.722590] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1878.040656] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1878.044842] Lustre: Skipped 1 previous similar message [ 1880.142543] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1880.146328] Lustre: Skipped 2 previous similar messages [ 1884.702461] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1884.707345] Lustre: Skipped 6 previous similar messages [ 1892.777758] Lustre: Modifying parameter general.jobid_var=TEST_123H in log params [ 1892.785685] Lustre: Skipped 10 previous similar messages [ 1908.883545] Lustre: Modifying parameter general.jobid_var=jobid_var=disable in log params [ 1908.888544] Lustre: Skipped 24 previous similar messages [ 1945.360807] Lustre: 87594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/jobid_var=jobid_var=disable (mode = 0) failed: rc = -17 [ 1945.866784] Lustre: Disabling parameter general.jobid_var= in log params [ 1945.872594] Lustre: Skipped 50 previous similar messages [ 1947.121432] Lustre: DEBUG MARKER: == conf-sanity test 124: check failover after replace_nids ========================================================== 05:22:16 (1789636936) [ 1948.358716] Lustre: DEBUG MARKER: SKIP: conf-sanity test_124 needs MDT failover setup [ 1949.555870] Lustre: DEBUG MARKER: == conf-sanity test 126: mount in parallel shouldn't cause a crash ========================================================== 05:22:18 (1789636938) [ 1953.760900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1953.769622] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1953.777547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.786063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1954.789251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1954.790980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1954.812511] Lustre: Skipped 1 previous similar message [ 1958.480919] Lustre: server umount lustre-OST0000 complete [ 1964.513647] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1964.519281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1964.533080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1966.853781] Lustre: server umount lustre-MDT0000 complete [ 1969.697365] LustreError: 87596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789636958 with bad export cookie 4667117784020749220 [ 1969.699210] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1969.704496] LustreError: 87596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1969.960829] Lustre: server umount lustre-MDT0001 complete [ 1977.656949] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 1979.601600] Key type lgssc unregistered [ 1979.810585] LNet: 88841:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1979.823194] LNetError: 88841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1979.840438] LNet: Removed LNI 192.168.203.139@tcp [ 1980.385224] Key type .llcrypt unregistered [ 1980.388576] Key type ._llcrypt unregistered [ 1983.308348] Key type ._llcrypt registered [ 1983.310215] Key type .llcrypt registered [ 1983.378367] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module ../libcfs/libcfs/libcfs [ 1986.096283] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules [ 1986.907840] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1986.963825] alg: No test for adler32 (adler32-zlib) [ 1987.956591] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 1988.141691] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 1989.783539] Key type lgssc registered [ 1990.440473] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1991.169528] LustreError: 90039:0:(lu_object.c:1468:lu_context_key_register()) cfs_race id 60d sleeping [ 2001.801619] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2007.213343] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2012.990345] LustreError: 90039:0:(lu_object.c:1468:lu_context_key_register()) cfs_fail_race id 60d awake: rc=0 [ 2013.006656] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2014.444878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2017.557986] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2023.137082] Lustre: DEBUG MARKER: == conf-sanity test 127: direct io overwrite on full ost ========================================================== 05:23:32 (1789637012) [ 2024.928244] LustreError: 91063:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.938395] LustreError: 91063:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2026.258107] LustreError: 91094:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2026.264892] LustreError: 91094:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2026.269907] LustreError: 91094:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 12, retries 0, failed: rc = -5 [ 2026.454812] Lustre: server umount lustre-MDT0000 complete [ 2034.070855] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 2035.797357] Key type lgssc unregistered [ 2035.963813] LNet: 92587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2035.968347] LNetError: 92587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2035.985700] LNet: Removed LNI 192.168.203.139@tcp [ 2036.516262] Key type .llcrypt unregistered [ 2036.517563] Key type ._llcrypt unregistered [ 2048.176657] Key type ._llcrypt registered [ 2048.178742] Key type .llcrypt registered [ 2048.290635] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2049.004638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.014311] alg: No test for adler32 (adler32-zlib) [ 2050.050057] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 2050.356132] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 2052.007203] Key type lgssc registered [ 2052.821177] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2057.903889] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2057.917849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2059.206544] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2061.944818] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2066.950094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.309791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2069.820141] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2074.422931] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2077.667532] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2081.915572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2082.340727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2086.332733] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2091.590557] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2128.658153] Lustre: DEBUG MARKER: == conf-sanity test 128: Force using remote logs with --nolocallogs ========================================================== 05:25:18 (1789637118) [ 2129.800189] Lustre: DEBUG MARKER: SKIP: conf-sanity test_128 need separate mgs device [ 2131.035804] Lustre: DEBUG MARKER: == conf-sanity test 129: attempt to connect an OST with the same index should fail ========================================================== 05:25:20 (1789637120) [ 2133.984463] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2133.993238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2134.005679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.597668] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.598562] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2138.606016] Lustre: Skipped 1 previous similar message [ 2138.615271] Lustre: Skipped 2 previous similar messages [ 2139.852500] Lustre: server umount lustre-MDT0000 complete [ 2142.675703] LustreError: 94521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637131 with bad export cookie 2476335851996964980 [ 2142.679453] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2142.687077] LustreError: 94521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2142.938154] Lustre: server umount lustre-MDT0001 complete [ 2156.095378] Lustre: server umount lustre-OST0000 complete [ 2162.715075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2162.972830] LustreError: 98052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2162.985189] LustreError: 98052:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2163.066461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2165.459513] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2168.289615] LustreError: 98053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2168.299566] LustreError: 98053:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2169.648524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2170.057351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2172.528926] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2176.895829] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2180.210582] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2188.422837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2188.515089] LustreError: 100142:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2188.586475] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2188.591885] LustreError: 98040:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2188.615498] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2188.621689] LustreError: 100135:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2188.624977] LustreError: 100135:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2188.627891] LustreError: 100135:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2188.692505] Lustre: server umount lustre-OST0000 complete [ 2188.698221] LustreError: 100135:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2193.885165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2193.987115] LustreError: Server lustre-OST0000 requested index 0, but that index is already in use. Use --writeconf to force [ 2193.992382] LustreError: 98040:0:(mgs_handler.c:605:mgs_target_reg()) Failed to write lustre-OST0000 log (-98) [ 2193.999983] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -98. Please see messages on the MGS. [ 2194.005634] LustreError: 100596:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -98 [ 2194.010880] LustreError: 100596:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 2194.013967] LustreError: 100596:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 2194.051759] Lustre: server umount lustre-OST0000 complete [ 2194.054338] LustreError: 100596:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -98 [ 2199.379381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2199.472241] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2199.475231] Lustre: Found index 0 for lustre-OST0000, updating log [ 2199.479557] Lustre: Client log for lustre-OST0000 was not updated; writeconf the MDT first to regenerate it. [ 2199.528959] Lustre: lustre-OST0000: new disk, initializing [ 2199.531680] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2199.536621] Lustre: 101131:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2199.678624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2203.923646] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2206.257130] Lustre: Failing over lustre-OST0000 [ 2206.346726] Lustre: server umount lustre-OST0000 complete [ 2208.655884] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2208.657880] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2208.664572] Lustre: Skipped 1 previous similar message [ 2208.672101] Lustre: Skipped 1 previous similar message [ 2213.858813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2213.861884] Lustre: Skipped 1 previous similar message [ 2214.861586] Lustre: server umount lustre-MDT0000 complete [ 2214.882053] LustreError: 98049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.891642] LustreError: 98049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2216.848577] LustreError: 98033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637205 with bad export cookie 2476335851996967724 [ 2216.852885] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2216.855974] LustreError: 98033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2217.019840] Lustre: server umount lustre-MDT0001 complete [ 2221.746174] Lustre: DEBUG MARKER: == conf-sanity test 130: re-register an MDT after writeconf ========================================================== 05:26:51 (1789637211) [ 2232.062526] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2237.802336] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2237.996719] LustreError: 103473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.046820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2240.141312] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2244.179530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2246.700257] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2248.754697] Lustre: Setting parameter general.debug_raw_pointers=Y in log params [ 2252.839133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2256.247839] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2260.855285] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2260.896284] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 2260.941654] Lustre: lustre-OST0001: new disk, initializing [ 2260.945307] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2260.949213] Lustre: 105791:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2264.484206] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2268.701153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2268.706595] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2268.782679] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2269.902242] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2272.121896] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 2274.271882] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 2274.274935] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2274.280897] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2278.885383] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2278.891897] Lustre: Skipped 1 previous similar message [ 2280.259855] Lustre: server umount lustre-MDT0001 complete [ 2281.528747] LustreError: 103468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.539196] LustreError: 103468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2284.813915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.892273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2284.896272] Lustre: Found index 1 for lustre-MDT0001, updating log [ 2284.922460] Lustre: 103461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2285.244938] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2285.248291] Lustre: Skipped 3 previous similar messages [ 2286.055951] LustreError: 107372:0:(obd_mount.c:234:lustre_start_simple()) lustre-MDT0001-lwp-OST0001 attach error -17 [ 2286.059538] LustreError: 107372:0:(tgt_mount.c:675:lustre_lwp_setup()) lustre-MDT0001-lwp-OST0001: setup up failed: rc -17 [ 2286.064575] LustreError: 103465:0:(mgc_request.c:614:do_requeue()) failed processing log: -17 [ 2286.123400] LustreError: 107377:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.203.139@tcp: cfg command failed: rc = -17 [ 2286.130594] Lustre: cmd=cf001 0:lustre-MDT0001-osp-MDT0000 1:osp 2:lustre-MDT0000-mdtlov_UUID [ 2286.130594] [ 2287.275967] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2290.666733] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2292.369112] Lustre: DEBUG MARKER: == conf-sanity test 131: MDT backup restore with project ID and inheritance flag ========================================================== 05:28:02 (1789637282) [ 2301.322749] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2310.423370] Lustre: 109398:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2322.032953] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2323.918426] Lustre: 110531:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2387.424078] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2387.427417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2387.431994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2387.434315] Lustre: Skipped 4 previous similar messages [ 2389.923300] Lustre: server umount lustre-MDT0000 complete [ 2391.961961] LustreError: 103454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637381 with bad export cookie 2476335851996968648 [ 2391.964158] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2391.969846] LustreError: 103454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2392.172695] Lustre: server umount lustre-MDT0001 complete [ 2404.498365] Lustre: server umount lustre-OST0000 complete [ 2416.861141] Lustre: server umount lustre-OST0001 complete [ 2420.028377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2425.357572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2429.919534] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2434.552588] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 2444.541222] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2449.915434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2449.928891] Lustre: lustre-MDT0000: reset Object Index mappings [ 2450.152095] LustreError: 114476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2450.160780] LustreError: 114476:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2450.197404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2452.161531] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2456.351688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2456.369318] Lustre: lustre-MDT0001: reset Object Index mappings [ 2458.708306] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2460.271069] Lustre: 115623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 2463.991608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2466.478163] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:194 to 0x280000400:225) [ 2467.197525] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2471.308023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.325374] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2475.756778] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:355 to 0x280000401:386) [ 2475.759106] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:194 to 0x2c0000400:225) [ 2475.760482] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:162 to 0x2c0000401:193) [ 2483.635348] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2485.587667] Lustre: 117499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 2514.235951] Lustre: DEBUG MARKER: == conf-sanity test 132: hsm_actions processed after failover ========================================================== 05:31:43 (1789637503) [ 2542.563341] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2542.566050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2542.575439] Lustre: Skipped 6 previous similar messages [ 2542.580351] Lustre: Skipped 5 previous similar messages [ 2547.841601] Lustre: server umount lustre-MDT0000 complete [ 2549.455487] LustreError: 117503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789637538 with bad export cookie 2476335851997015436 [ 2549.461353] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2549.464418] LustreError: 117503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2552.802882] LustreError: 114473:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.813285] LustreError: 114473:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2555.805784] Lustre: server umount lustre-MDT0001 complete [ 2563.678580] Lustre: server umount lustre-OST0000 complete [ 2572.020674] Lustre: server umount lustre-OST0001 complete [ 2575.626914] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 2581.378394] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2600.795033] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2600.920343] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2600.938660] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2600.983388] Lustre: lustre-MDT0000: new disk, initializing [ 2601.021144] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2601.024421] Lustre: Skipped 3 previous similar messages [ 2601.031153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2602.752365] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2606.227281] Lustre: server umount lustre-MDT0000 complete [ 2610.602710] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.747591] Lustre: Found index 0 for lustre-MDT0000, updating log [ 2610.754379] Lustre: 121777:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2610.763386] Lustre: Setting parameter lustre-MDT0000.mdt.hsm_control=enabled in log lustre-MDT0000 [ 2612.843882] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2615.342601] Lustre: server umount lustre-MDT0000 complete [ 2619.755148] Lustre: DEBUG MARKER: == conf-sanity test 133: stripe QOS: free space balance in a pool ========================================================== 05:33:29 (1789637609) [ 2620.517748] Lustre: DEBUG MARKER: SKIP: conf-sanity test_133 needs >= 4 OSTs [ 2621.306138] Lustre: DEBUG MARKER: == conf-sanity test 134: check_iam works without faults == 05:33:31 (1789637611) [ 2629.677329] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 2634.330758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2636.288821] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2640.014783] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2640.068929] Lustre: 123827:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2640.100396] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2640.103121] Lustre: Skipped 1 previous similar message [ 2640.143043] Lustre: lustre-MDT0001: new disk, initializing [ 2640.194892] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2640.199492] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2641.751427] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2644.359545] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 2647.383775] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2647.553100] Lustre: lustre-OST0000: new disk, initializing [ 2647.556634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2647.563032] Lustre: 125404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2649.167287] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2649.172850] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2649.234739] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2650.238643] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2655.350802] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2655.430582] Lustre: 126283:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2657.493442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2657.498978] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2657.547715] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2657.928576] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 2663.430755] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 2664.937885] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 3091.120823] Lustre: 132702:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [ 3091.134228] Lustre: 132702:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [ 3091.142147] Lustre: 132702:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [ 3091.149800] Lustre: 132702:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [ 3091.155288] Lustre: 132702:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [ 3091.167099] Lustre: 132702:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 3458.220867] Lustre: DEBUG MARKER: == conf-sanity test 135: check the behavior when changelog is wrapped around ========================================================== 05:47:26 (1789638446) [ 3475.433908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3475.445125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.459091] Lustre: Skipped 2 previous similar messages [ 3475.465778] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3475.473762] Lustre: Skipped 8 previous similar messages [ 3480.133621] Lustre: server umount lustre-MDT0000 complete [ 3482.087354] LustreError: 127102:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3482.108927] LustreError: 127102:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 3484.152283] LustreError: 124565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789638473 with bad export cookie 2476335851997076581 [ 3484.159265] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3484.164988] LustreError: 124565:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3487.201488] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3487.217832] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3487.226809] Lustre: Skipped 4 previous similar messages [ 3487.243965] Lustre: Skipped 5 previous similar messages [ 3490.671219] Lustre: server umount lustre-MDT0001 complete [ 3509.215160] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3509.389527] Lustre: server umount lustre-OST0000 complete [ 3519.658515] Lustre: server umount lustre-OST0001 complete [ 3526.520478] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 3537.078313] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 3572.852861] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3573.201527] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3573.233291] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3573.239202] Lustre: Skipped 1 previous similar message [ 3573.327942] Lustre: lustre-MDT0000: new disk, initializing [ 3573.330958] Lustre: Skipped 1 previous similar message [ 3573.413373] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.421480] Lustre: Skipped 5 previous similar messages [ 3573.447751] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3578.727688] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3590.036700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3590.214117] Lustre: 135781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3590.346125] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3590.356575] Lustre: Skipped 1 previous similar message [ 3590.472083] Lustre: lustre-MDT0001: new disk, initializing [ 3590.672289] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3590.691858] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3596.082950] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3606.121693] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3611.147771] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3617.201629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3617.521539] Lustre: lustre-OST0000: new disk, initializing [ 3617.526206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3617.534308] Lustre: 137749:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3617.617207] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.621369] Lustre: Skipped 1 previous similar message [ 3618.821351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3618.834910] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3618.964412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3623.778250] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 3632.792227] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3643.141071] Lustre: lustre-MDD0000: changelog on [ 3645.889891] Lustre: lustre-MDD0001: changelog on [ 5732.125695] Lustre: lustre-MDD0001: changelog off [ 5734.242496] Lustre: 143037:0:(llog_cat.c:310:llog_cat_process_or_fork()) lustre-MDD0000: catlog [0xa:0x5:0x0] crosses index zero [ 5735.147353] Lustre: lustre-MDD0000: changelog off [ 5743.587130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5743.592937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5743.606028] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5743.617761] Lustre: Skipped 1 previous similar message [ 5743.632666] Lustre: Skipped 3 previous similar messages [ 5748.060883] Lustre: server umount lustre-OST0000 complete [ 5753.824747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5753.851165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5753.865209] Lustre: Skipped 1 previous similar message [ 5758.244584] Lustre: server umount lustre-MDT0000 complete [ 5758.947567] LustreError: 135788:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5758.971030] LustreError: 135788:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 5761.727828] LustreError: 135775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640750 with bad export cookie 2476335851997899417 [ 5761.728401] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5761.739556] LustreError: 135775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5762.053593] Lustre: server umount lustre-MDT0001 complete [ 5773.135291] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 5776.492562] Key type lgssc unregistered [ 5776.820402] LNet: 146922:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5776.832062] LNetError: 146922:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5776.889961] LNet: Removed LNI 192.168.203.139@tcp [ 5777.755174] Key type .llcrypt unregistered [ 5777.757409] Key type ._llcrypt unregistered [ 5800.190537] Key type ._llcrypt registered [ 5800.192419] Key type .llcrypt registered [ 5800.284738] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5815.578443] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 5816.355894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5816.418977] alg: No test for adler32 (adler32-zlib) [ 5817.524911] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 5817.729441] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 5819.375207] Key type lgssc registered [ 5820.440555] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5853.864414] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5853.890625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5855.165733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5855.205368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5855.315634] Lustre: lustre-MDT0000: new disk, initializing [ 5855.408397] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5855.437859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5860.147727] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5870.074322] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5870.197370] Lustre: 150120:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5870.262839] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5870.272634] Lustre: Skipped 1 previous similar message [ 5870.336694] Lustre: lustre-MDT0001: new disk, initializing [ 5870.428694] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5870.491187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5870.498932] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5874.897807] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5884.172920] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5890.179654] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5896.315880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5896.559484] Lustre: lustre-OST0000: new disk, initializing [ 5896.561752] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5896.566302] Lustre: 152089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5896.638763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5901.963345] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 5906.486076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5906.495195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5906.626464] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5910.812678] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5917.000793] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_module obdecho/obdecho [ 5918.549317] LustreError: 153091:0:(echo_client.c:671:echo_device_alloc()) lustre-MDT0001 no type name for echo mdt setup [ 5918.555551] LustreError: 153091:0:(obd_config.c:845:class_setup()) setup ec failed (-22) [ 5920.291779] LustreError: 153187:0:(echo_client.c:1884:echo_md_handler()) Can not resolve the path : rc = -2 [ 5932.517965] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.532146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5932.552317] Lustre: Skipped 2 previous similar messages [ 5932.555480] Lustre: Skipped 1 previous similar message [ 5937.646864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5937.657532] Lustre: Skipped 1 previous similar message [ 5938.732277] Lustre: server umount lustre-MDT0000 complete [ 5942.539173] LustreError: 150111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789640931 with bad export cookie 17278866440995382116 [ 5942.540263] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5942.548368] LustreError: 150111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5942.756381] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5942.892911] Lustre: server umount lustre-MDT0001 complete [ 5948.686365] Lustre: server umount lustre-OST0000 complete [ 5972.179597] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 5985.875693] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6021.079952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6021.277677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6021.318666] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6021.411145] Lustre: lustre-MDT0000: new disk, initializing [ 6021.487127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6021.503572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6026.419280] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6037.992836] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6038.111992] Lustre: 156924:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6038.165574] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6038.169405] Lustre: Skipped 1 previous similar message [ 6038.281991] Lustre: lustre-MDT0001: new disk, initializing [ 6038.404147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6038.442063] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6038.454054] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6043.405946] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6052.133140] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6056.967173] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6062.631462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6062.928394] Lustre: lustre-OST0000: new disk, initializing [ 6062.932223] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6062.939513] Lustre: 158892:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6063.040029] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6068.773861] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6072.919062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6072.931435] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6073.031384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6076.989185] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6081.514711] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6081.686488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6085.451339] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6085.609202] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6088.677713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6088.680973] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6088.698143] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6088.716043] Lustre: Skipped 1 previous similar message [ 6088.727572] Lustre: Skipped 2 previous similar messages [ 6093.355981] Lustre: server umount lustre-OST0000 complete [ 6098.913289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6098.917353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6098.922051] Lustre: Skipped 1 previous similar message [ 6104.038069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6104.051247] Lustre: Skipped 2 previous similar messages [ 6111.199128] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6111.313961] Lustre: server umount lustre-MDT0000 complete [ 6113.253275] LustreError: 156928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6113.268318] LustreError: 156928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 6114.460025] LustreError: 156914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641103 with bad export cookie 17278866440995383075 [ 6114.461567] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6114.467963] LustreError: 156914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6114.748764] Lustre: server umount lustre-MDT0001 complete [ 6117.995932] Lustre: DEBUG MARKER: == conf-sanity test 137: a new MDT should inherit pools, parameters and failnode ========================================================== 06:31:47 (1789641107) [ 6119.215171] Lustre: DEBUG MARKER: SKIP: conf-sanity test_137 needs >= 3 MDTs [ 6120.683804] Lustre: DEBUG MARKER: == conf-sanity test 140: remove_updatelog script actions ========================================================== 06:31:49 (1789641109) [ 6126.532413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.872058] LustreError: 161303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6126.920082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6130.274264] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6132.193286] LustreError: 161304:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6136.291066] LustreError: 161303:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.306748] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6141.194193] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6147.520466] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6152.243720] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6158.358793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6164.594422] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6170.646237] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6182.160605] Lustre: server umount lustre-MDT0000 complete [ 6182.375547] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6182.390161] LustreError: 161300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6182.400560] Lustre: Skipped 2 previous similar messages [ 6182.417825] LustreError: 161300:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6185.580816] LustreError: 162038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641174 with bad export cookie 17278866440995383957 [ 6185.584409] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6185.595613] LustreError: 162038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6186.603224] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [ 6186.608808] Lustre: Skipped 2 previous similar messages [ 6190.497057] LustreError: 161299:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6190.526370] LustreError: 161299:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 6192.120230] Lustre: server umount lustre-MDT0001 complete [ 6198.823530] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 6211.641938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6211.985658] LustreError: 165142:0:(llog.c:1655:llog_backup()) MGC192.168.203.139@tcp: failed to open log lustre-sptlrpc: rc = -108 [ 6211.998759] Lustre: 165142:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-sptlrpc: rc = -108 [ 6215.137080] LustreError: 165147:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6215.163140] LustreError: 165147:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6247.492057] LustreError: 165165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6247.520918] LustreError: 165165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 6252.601862] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (not set up) [ 6252.612201] Lustre: Skipped 3 previous similar messages [ 6272.992368] LustreError: Failed to get MGS log lustre-mdtir and no local copy. [ 6273.003719] Lustre: 165142:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.203.139@tcp: IR log lustre-mdtir failed, not fatal: rc = -2 [ 6311.995996] LustreError: 165149:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6312.021669] LustreError: 165149:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 40 previous similar messages [ 6313.998320] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6314.006464] Lustre: Skipped 2 previous similar messages [ 6317.771756] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6319.086198] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6327.105742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.002397] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 6336.704836] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6345.666432] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6348.102552] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6355.019770] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6356.930592] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6376.934085] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6376.937726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6376.939896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6376.950825] Lustre: Skipped 2 previous similar messages [ 6376.988114] LustreError: Skipped 1 previous similar message [ 6382.510815] Lustre: server umount lustre-MDT0000 complete [ 6385.789086] LustreError: 162038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641374 with bad export cookie 17278866440995390411 [ 6385.790145] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6385.813336] LustreError: 162038:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6386.186119] Lustre: server umount lustre-MDT0001 complete [ 6399.345690] Lustre: server umount lustre-OST0000 complete [ 6420.802122] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6430.637525] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6467.012779] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6467.278807] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6467.303499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6467.403536] Lustre: lustre-MDT0000: new disk, initializing [ 6467.459778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.465498] Lustre: Skipped 1 previous similar message [ 6467.477544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6471.965031] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6482.942657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6483.053852] Lustre: 170248:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6483.398410] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6487.596334] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6496.311775] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6501.198749] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6506.834621] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6507.006728] Lustre: lustre-OST0000: new disk, initializing [ 6507.008394] Lustre: Skipped 1 previous similar message [ 6507.017889] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6507.020979] Lustre: Skipped 2 previous similar messages [ 6507.024285] Lustre: 172217:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6511.703318] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6514.816085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6514.833459] Lustre: Skipped 1 previous similar message [ 6514.837599] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6514.951189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6521.607524] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6526.461330] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6526.589435] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6532.108171] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6532.273579] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6535.651617] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6535.654721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6535.669859] Lustre: Skipped 1 previous similar message [ 6535.690033] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6535.692902] Lustre: Skipped 5 previous similar messages [ 6540.029612] Lustre: server umount lustre-OST0000 complete [ 6544.873759] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6549.632768] Lustre: server umount lustre-MDT0000 complete [ 6551.008162] LustreError: 171510:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6551.030852] LustreError: 171510:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 6552.740049] LustreError: 170242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641541 with bad export cookie 17278866440995407239 [ 6552.742231] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6552.752864] LustreError: 170242:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6553.018402] Lustre: server umount lustre-MDT0001 complete [ 6560.904409] Lustre: DEBUG MARKER: == conf-sanity test 150: test setting max_cached_mb to a % ========================================================== 06:39:10 (1789641550) [ 6566.848140] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6571.803243] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6580.230734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.947100] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6592.869524] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6598.354373] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6605.327607] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6605.703509] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.707566] Lustre: Skipped 4 previous similar messages [ 6612.132641] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6619.374886] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6630.150555] Lustre: DEBUG MARKER: == conf-sanity test 151a: damaged local config doesn't prevent mounting ========================================================== 06:40:18 (1789641618) [ 6637.546573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6637.562969] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6637.584393] Lustre: Skipped 3 previous similar messages [ 6641.100199] Lustre: server umount lustre-OST0000 complete [ 6647.277964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6647.292946] LustreError: Skipped 1 previous similar message [ 6651.474875] Lustre: server umount lustre-MDT0000 complete [ 6655.380265] LustreError: 174610:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641644 with bad export cookie 17278866440995408121 [ 6655.386820] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6655.699532] Lustre: server umount lustre-MDT0001 complete [ 6665.417997] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 6668.014853] Key type lgssc unregistered [ 6668.240562] LNet: 178356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6668.254778] LNetError: 178356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6668.283723] LNet: Removed LNI 192.168.203.139@tcp [ 6669.070984] Key type .llcrypt unregistered [ 6669.076352] Key type ._llcrypt unregistered [ 6687.189121] Key type ._llcrypt registered [ 6687.192943] Key type .llcrypt registered [ 6687.296364] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6688.580081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6688.647465] alg: No test for adler32 (adler32-zlib) [ 6689.769321] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 6690.009805] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 6691.703241] Key type lgssc registered [ 6692.786701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6700.692836] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6700.755125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6716.383454] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6721.504336] LustreError: 179597:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.139@tcp: failed processing log, type 4: rc = -110 [ 6726.624358] LustreError: 179597:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6726.644264] LustreError: Failed to get MGS log lustre-OST0000 and no local copy. [ 6726.649559] LustreError: MGC192.168.203.139@tcp: Configuration from log lustre-OST0000 failed from MGS -22. Check client and MGS are on compatible version. [ 6726.664742] LustreError: 179597:0:(tgt_mount.c:1884:server_start_targets()) failed to start server lustre-OST0000: -22 [ 6726.674941] LustreError: 179597:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -22 [ 6726.685747] LustreError: 179597:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 6726.729900] Lustre: server umount lustre-OST0000 complete [ 6726.735495] LustreError: 179597:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 6734.883266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6736.526702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6741.348949] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6746.083325] LustreError: 180098:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6746.100932] LustreError: 180098:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6750.646367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6751.009885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6755.830675] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6763.835183] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6770.204609] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6777.605951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6777.842137] LustreError: 181914:0:(llog_osd.c:246:llog_osd_read_header()) lustre-OST0000-osd: bad log lustre-OST0000 [0xa:0x6:0x0] header magic: 0x0 (expected 0x10645539) [ 6777.850269] Lustre: 181914:0:(mgc_request_server.c:750:mgc_llog_local_copy()) MGC192.168.203.139@tcp: invalid local config log lustre-OST0000: rc = -22 [ 6777.987284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6785.770401] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6794.661941] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6797.281866] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6797.288187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6797.313247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6800.871124] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6800.873510] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6800.883183] Lustre: Skipped 1 previous similar message [ 6800.887047] Lustre: Skipped 1 previous similar message [ 6803.365727] Lustre: server umount lustre-MDT0000 complete [ 6805.989775] LustreError: 180092:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6806.034363] LustreError: 180092:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6807.808587] LustreError: 180079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641796 with bad export cookie 14837657270664371804 [ 6807.815177] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6807.822155] LustreError: 180079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6808.623145] Lustre: server umount lustre-MDT0001 complete [ 6823.801962] Lustre: server umount lustre-OST0000 complete [ 6836.189699] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 6848.687376] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 6883.616895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6883.811753] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6883.839404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6883.880698] Lustre: lustre-MDT0000: new disk, initializing [ 6883.935930] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6883.957339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6887.608170] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6897.982278] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6898.138955] Lustre: 185439:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6898.224571] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6898.227578] Lustre: Skipped 1 previous similar message [ 6898.399175] Lustre: lustre-MDT0001: new disk, initializing [ 6898.535378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6898.581687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6898.594050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6903.374782] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6912.717260] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6918.420197] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6924.815162] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6925.024078] Lustre: lustre-OST0000: new disk, initializing [ 6925.029791] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6925.036423] Lustre: 187404:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6925.134658] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6931.318837] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 6935.090945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6935.097042] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6935.221479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6940.897417] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6945.990056] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6946.193869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6950.300947] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6950.484306] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6956.009922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6956.011069] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6956.012690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6956.012697] Lustre: Skipped 1 previous similar message [ 6956.024093] Lustre: Skipped 1 previous similar message [ 6956.033109] LustreError: Skipped 1 previous similar message [ 6958.583938] Lustre: server umount lustre-OST0000 complete [ 6965.217992] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6965.224684] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6965.234604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6965.244682] Lustre: Skipped 1 previous similar message [ 6968.687777] Lustre: server umount lustre-MDT0000 complete [ 6971.360365] LustreError: 185450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6971.368509] LustreError: 185450:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6972.068270] LustreError: 185430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789641961 with bad export cookie 14837657270664372525 [ 6972.072112] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6972.075938] LustreError: 185430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6972.390883] Lustre: server umount lustre-MDT0001 complete [ 6982.554672] Lustre: DEBUG MARKER: == conf-sanity test 151b: -ENOSPC doesn't affect mount === 06:46:10 (1789641970) [ 7001.578092] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 7003.877216] Key type lgssc unregistered [ 7004.172640] LNet: 190348:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7004.191030] LNetError: 190348:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7004.208584] LNet: Removed LNI 192.168.203.139@tcp [ 7005.163329] Key type .llcrypt unregistered [ 7005.165347] Key type ._llcrypt unregistered [ 7026.938651] Key type ._llcrypt registered [ 7026.941905] Key type .llcrypt registered [ 7027.088701] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7028.681488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7028.712318] alg: No test for adler32 (adler32-zlib) [ 7029.822980] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 7030.088898] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 7031.799184] Key type lgssc registered [ 7032.843544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7041.015289] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7041.045481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7042.555314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7047.244498] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7052.768303] LustreError: 191564:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7052.785431] LustreError: 191564:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7056.798091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7057.349857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7062.505481] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7070.586463] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7077.232216] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7085.229016] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7085.445844] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7085.453828] LustreError: 193428:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7085.474146] Lustre: 193428:0:(mgc_request_server.c:747:mgc_llog_local_copy()) MGC192.168.203.139@tcp: can't backup local config lustre-OST0000: rc = -28 [ 7085.602216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.517932] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7100.216946] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7103.456365] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7103.463149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7103.485834] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.041021] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.056060] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7108.960190] Lustre: server umount lustre-MDT0000 complete [ 7112.164108] LustreError: 193560:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7112.195196] LustreError: 193560:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7112.780291] LustreError: 191544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642101 with bad export cookie 3684154947312974198 [ 7112.795605] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.265885] Lustre: server umount lustre-MDT0001 complete [ 7127.645787] Lustre: server umount lustre-OST0000 complete [ 7137.146586] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7147.880233] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7187.190517] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7187.429216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7187.464630] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7187.568593] Lustre: lustre-MDT0000: new disk, initializing [ 7187.687171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.707710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7192.094115] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7203.319977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7203.453087] Lustre: 196955:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7203.494453] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7203.501537] Lustre: Skipped 1 previous similar message [ 7203.504252] LustreError: 197736:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-MDT0001: rc = -28 [ 7203.510179] LustreError: 197736:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7203.513340] Lustre: 197736:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-MDT0001: rc = -28 [ 7203.529840] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7203.533528] Lustre: Skipped 1 previous similar message [ 7203.585533] Lustre: lustre-MDT0001: new disk, initializing [ 7203.762880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7203.794387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7203.813917] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7208.317973] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7217.702972] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7224.159836] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7231.800364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7232.212949] Lustre: *** cfs_fail_loc=131e, val=0*** [ 7232.222543] Lustre: Skipped 1 previous similar message [ 7232.228643] LustreError: 198899:0:(llog.c:1696:llog_backup()) MGC192.168.203.139@tcp: failed to backup log lustre-OST0000: rc = -28 [ 7232.242933] LustreError: 198899:0:(llog.c:1696:llog_backup()) Skipped 1 previous similar message [ 7232.252792] Lustre: 198899:0:(mgc_request_server.c:770:mgc_llog_local_copy()) MGC192.168.203.139@tcp: failed to copy new config lustre-OST0000: rc = -28 [ 7232.266145] Lustre: 198899:0:(mgc_request_server.c:770:mgc_llog_local_copy()) Skipped 1 previous similar message [ 7232.303546] Lustre: lustre-OST0000: new disk, initializing [ 7232.311724] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7232.322768] Lustre: 198923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7232.461464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7238.752332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7238.774482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7238.902813] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7239.760497] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7249.225978] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7253.515896] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7253.666854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7257.486949] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7257.700924] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7264.740670] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7264.747707] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7264.753062] LustreError: Skipped 1 previous similar message [ 7264.768053] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7264.782943] Lustre: Skipped 2 previous similar messages [ 7264.804721] Lustre: Skipped 3 previous similar messages [ 7265.998659] Lustre: server umount lustre-OST0000 complete [ 7274.985929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7274.999040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7275.005239] Lustre: Skipped 1 previous similar message [ 7275.023557] Lustre: Skipped 1 previous similar message [ 7280.110361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7280.118176] Lustre: Skipped 1 previous similar message [ 7285.215279] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7285.681813] Lustre: server umount lustre-MDT0000 complete [ 7289.313407] LustreError: 196965:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7289.338665] LustreError: 196965:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7290.347061] LustreError: 196945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642279 with bad export cookie 3684154947312974919 [ 7290.362857] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7290.378100] LustreError: 196945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7290.824863] Lustre: server umount lustre-MDT0001 complete [ 7301.177230] Lustre: DEBUG MARKER: == conf-sanity test 152: seq allocation error in OSP ===== 06:51:29 (1789642289) [ 7320.814679] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7331.333611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7331.815314] LustreError: 202053:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7331.899985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7336.927836] LustreError: 202054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7336.981145] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7346.157775] LustreError: 202054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7346.174605] LustreError: 202054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7346.206103] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.865933] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7354.874892] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 7362.100431] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7368.939443] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7378.168679] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7378.325203] Lustre: lustre-OST0001: new disk, initializing [ 7378.329658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7378.334747] Lustre: 204359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7378.438920] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7378.453352] Lustre: Skipped 2 previous similar messages [ 7385.181524] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7385.672501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7385.683797] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7385.782689] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7396.666987] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 7405.042311] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 7407.373658] Lustre: DEBUG MARKER: ADD OST3 [ 7413.382426] loop: module loaded [ 7413.488905] loop0: detected capacity change from 0 to 204800000 [ 7413.798937] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7413.824183] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7413.831519] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 7421.704246] Lustre: DEBUG MARKER: STOP OST3 [ 7421.711110] loop0: detected capacity change from 0 to 204800000 [ 7429.161601] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7429.595328] Lustre: lustre-OST0003: new disk, initializing [ 7429.608484] Lustre: srv-lustre-OST0003: No data found on store. Initialize space. [ 7429.629201] Lustre: 206405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7431.065480] Lustre: *** cfs_fail_loc=2109, val=2*** [ 7431.371150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3:ost [ 7431.392533] Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3:ost] [ 7431.460288] Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x300000400 [ 7433.130400] LustreError: 206498:0:(osp_precreate.c:1285:osp_precreate_thread()) lustre-OST0003-osc-MDT0001: init pre fid error: rc = -5 [ 7434.175877] Lustre: Failing over lustre-OST0003 [ 7434.607295] Lustre: server umount lustre-OST0003 complete [ 7436.295523] Lustre: lustre-OST0003-osc-MDT0001: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7436.335460] LustreError: 203537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7436.379475] LustreError: 203537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7439.374924] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7449.532946] Lustre: DEBUG MARKER: START OST3 again [ 7456.766933] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7457.129422] Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180 [ 7457.134226] Lustre: Skipped 1 previous similar message [ 7457.155233] Lustre: lustre-OST0003: in recovery but waiting for the first client to connect [ 7458.408163] Lustre: lustre-OST0003: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 7458.822504] Lustre: lustre-OST0003: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 7458.828989] Lustre: lustre-OST0003-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7462.470577] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7487.412331] Lustre: Failing over lustre-OST0003 [ 7487.809702] Lustre: server umount lustre-OST0003 complete [ 7488.006261] Lustre: lustre-OST0003-osc-MDT0000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7488.023122] LustreError: 203539:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0003: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7488.031612] Lustre: Skipped 2 previous similar messages [ 7488.086623] LustreError: 203539:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 14 previous similar messages [ 7510.496795] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7510.505154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7510.532666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7510.540207] Lustre: Skipped 2 previous similar messages [ 7515.010994] Lustre: server umount lustre-MDT0000 complete [ 7518.972486] LustreError: 202034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642508 with bad export cookie 3684154947312975801 [ 7518.979991] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7518.984712] LustreError: 202034:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7525.650338] Lustre: server umount lustre-MDT0001 complete [ 7530.534542] Lustre: server umount lustre-OST0000 complete [ 7534.282664] Lustre: server umount lustre-OST0001 complete [ 7540.949479] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7552.305545] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7600.283957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7600.743609] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7600.785845] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7600.901302] Lustre: lustre-MDT0000: new disk, initializing [ 7601.034823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7601.079334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7605.784526] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7617.080183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7617.218286] Lustre: 211822:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7617.298426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7617.302147] Lustre: Skipped 1 previous similar message [ 7617.351694] Lustre: lustre-MDT0001: new disk, initializing [ 7617.480706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7617.496202] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7622.066079] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7632.328972] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7639.191410] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7646.420541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7646.782940] Lustre: 213796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7648.470494] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7648.640028] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7654.255699] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7665.695944] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7671.370450] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7671.620584] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7676.745157] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7676.976222] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7679.456727] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7679.465904] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7679.481409] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7679.492536] Lustre: Skipped 6 previous similar messages [ 7679.496686] Lustre: Skipped 6 previous similar messages [ 7693.279119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7693.537570] Lustre: server umount lustre-OST0000 complete [ 7699.427374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7699.435132] LustreError: Skipped 1 previous similar message [ 7703.766388] Lustre: server umount lustre-MDT0000 complete [ 7705.056491] LustreError: 212680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7705.073772] LustreError: 212680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 7707.796143] LustreError: 211816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642696 with bad export cookie 3684154947312978090 [ 7707.801624] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7707.806833] LustreError: 211816:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7708.408844] Lustre: server umount lustre-MDT0001 complete [ 7712.283695] Lustre: DEBUG MARKER: == conf-sanity test 153a: bypass invalid NIDs quickly ==== 06:58:21 (1789642701) [ 7730.127349] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 7740.381920] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 7776.938636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7777.115684] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7777.137732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7777.141062] Lustre: Skipped 1 previous similar message [ 7777.194284] Lustre: lustre-MDT0000: new disk, initializing [ 7777.197140] Lustre: Skipped 1 previous similar message [ 7777.264407] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7777.273338] Lustre: Skipped 1 previous similar message [ 7782.350856] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7792.980142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7793.131811] Lustre: 218408:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7793.473572] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7797.713817] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7808.066572] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7814.963763] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7823.513876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7823.846708] Lustre: 220378:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7825.585988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7825.751470] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7831.332172] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7841.406135] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7847.038282] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7847.337643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7852.158815] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7852.377645] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7856.611152] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7856.614289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7856.619888] LustreError: Skipped 1 previous similar message [ 7856.622406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7856.634456] Lustre: Skipped 3 previous similar messages [ 7856.639658] Lustre: Skipped 7 previous similar messages [ 7860.608498] Lustre: server umount lustre-OST0000 complete [ 7873.029256] Lustre: server umount lustre-MDT0000 complete [ 7876.070215] LustreError: 218419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7876.108132] LustreError: 218419:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7877.857312] LustreError: 218399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642866 with bad export cookie 3684154947312978972 [ 7877.869375] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7877.877121] LustreError: 218399:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7878.228286] Lustre: server umount lustre-MDT0001 complete [ 7888.620521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7889.411063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7889.414158] Lustre: Skipped 5 previous similar messages [ 7894.506669] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7903.749338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7909.155170] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7916.739464] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7923.130941] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7930.812922] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7938.067826] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 7946.865138] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7960.044883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7960.047879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7960.065992] Lustre: Skipped 6 previous similar messages [ 7964.622921] Lustre: server umount lustre-OST0000 complete [ 7976.107397] Lustre: server umount lustre-MDT0000 complete [ 7980.428090] LustreError: 222585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789642969 with bad export cookie 3684154947312979854 [ 7980.453605] LustreError: 222585:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7980.471961] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7980.848937] Lustre: server umount lustre-MDT0001 complete [ 8087.338409] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8091.968508] Key type lgssc unregistered [ 8092.469559] LNet: 226270:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8092.479647] LNetError: 226270:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8092.500916] LNet: Removed LNI 192.168.203.139@tcp [ 8094.031221] Key type .llcrypt unregistered [ 8094.033142] Key type ._llcrypt unregistered [ 8125.671318] Key type ._llcrypt registered [ 8125.678654] Key type .llcrypt registered [ 8125.812677] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8144.570673] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8146.444167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8146.483566] alg: No test for adler32 (adler32-zlib) [ 8147.704602] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 8148.148819] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8149.968593] Key type lgssc registered [ 8151.528615] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8196.174738] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8196.211851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8197.537874] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8197.576121] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8197.680944] Lustre: lustre-MDT0000: new disk, initializing [ 8197.841382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8197.852866] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8203.337182] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8213.743124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8213.874235] Lustre: 229633:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8213.928649] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8213.935864] Lustre: Skipped 1 previous similar message [ 8214.023167] Lustre: lustre-MDT0001: new disk, initializing [ 8214.166828] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8214.224975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8214.234379] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8219.164569] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8228.834306] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8234.402894] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8241.386139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8241.695465] Lustre: lustre-OST0000: new disk, initializing [ 8241.699831] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8241.709680] Lustre: 231601:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8241.806451] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8248.888664] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8250.957665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8250.966643] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8251.111585] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8258.530871] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8264.220799] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8264.489536] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8270.440220] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8270.663763] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8276.964412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8276.965860] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8276.973101] LustreError: Skipped 1 previous similar message [ 8276.980847] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8276.992139] Lustre: Skipped 1 previous similar message [ 8277.012379] Lustre: Skipped 1 previous similar message [ 8282.093036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8282.105598] Lustre: Skipped 1 previous similar message [ 8287.202648] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8287.203662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8287.230366] Lustre: Skipped 1 previous similar message [ 8287.501303] Lustre: server umount lustre-OST0000 complete [ 8292.324506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8292.334141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8292.346380] Lustre: Skipped 1 previous similar message [ 8292.356329] Lustre: Skipped 1 previous similar message [ 8297.441319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8297.454516] Lustre: Skipped 1 previous similar message [ 8297.605680] Lustre: server umount lustre-MDT0000 complete [ 8301.073126] LustreError: 229625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643290 with bad export cookie 13873411435442775120 [ 8301.086513] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8301.087244] LustreError: 229625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8301.479538] Lustre: server umount lustre-MDT0001 complete [ 8320.494978] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8330.589347] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8331.086848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8335.067997] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8336.352996] LustreError: 234537:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8336.365636] LustreError: 234537:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8341.477626] LustreError: 234538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8344.227799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8344.711971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8351.591523] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8355.822745] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 8363.427389] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8364.000377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8371.558829] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8381.077951] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8381.356407] Lustre: lustre-OST0001: new disk, initializing [ 8381.362070] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8381.373283] Lustre: 236845:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8381.545055] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8386.704136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8386.715609] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8386.870859] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8388.361889] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8399.459437] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 8406.348724] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 8421.685947] Lustre: DEBUG MARKER: == conf-sanity test 153c: don't stuck on unreached NID === 07:10:09 (1789643409) [ 8426.467039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8426.482478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8426.508112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8431.039595] Lustre: server umount lustre-MDT0000 complete [ 8433.126927] LustreError: 234533:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8433.159188] LustreError: 234533:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8434.701667] LustreError: 234517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643423 with bad export cookie 13873411435442776009 [ 8434.705679] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8434.715250] LustreError: 234517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8435.121936] Lustre: server umount lustre-MDT0001 complete [ 8451.076208] Lustre: server umount lustre-OST0000 complete [ 8454.111180] Lustre: 227854:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789643427/real 1789643427] req@ffff98203fb79f80 x1876576842139008/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789643443 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 8454.137441] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8454.148692] Lustre: Skipped 3 previous similar messages [ 8454.850507] Lustre: server umount lustre-OST0001 complete [ 8462.798213] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8473.636590] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8513.463916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8513.754710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8513.790543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8513.856954] Lustre: lustre-MDT0000: new disk, initializing [ 8513.936361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8513.988923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8518.135326] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8528.321871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8528.468447] Lustre: 240792:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8528.534247] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8528.538407] Lustre: Skipped 1 previous similar message [ 8528.607440] Lustre: lustre-MDT0001: new disk, initializing [ 8528.797730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8528.812336] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8534.115866] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8542.471382] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8548.292956] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8557.653930] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8558.054605] Lustre: lustre-OST0000: new disk, initializing [ 8558.059748] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8558.066117] Lustre: 242759:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8559.591481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8559.611144] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8559.735758] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8564.885436] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8574.742786] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8579.689635] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8579.887164] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8584.745977] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8585.042935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8590.316512] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8590.320438] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8590.327468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8590.327476] Lustre: Skipped 4 previous similar messages [ 8590.342873] Lustre: Skipped 1 previous similar message [ 8593.721916] Lustre: server umount lustre-OST0000 complete [ 8600.547635] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8600.577969] Lustre: Skipped 1 previous similar message [ 8600.585782] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8603.788991] Lustre: server umount lustre-MDT0000 complete [ 8605.669488] LustreError: 240804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8605.687982] LustreError: 240804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8607.709423] LustreError: 240785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643596 with bad export cookie 13873411435442778144 [ 8607.717061] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8607.720818] LustreError: 240785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8608.171589] Lustre: server umount lustre-MDT0001 complete [ 8617.497733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8618.011162] LustreError: 244978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8618.127779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8618.132454] Lustre: Skipped 2 previous similar messages [ 8622.968609] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8628.710334] LustreError: 244978:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8628.728079] LustreError: 244978:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8632.476734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.284723] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8645.118527] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8652.000373] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8660.616380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8668.190498] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 8675.635439] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8821.224986] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8821.233569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8821.238145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8821.246359] Lustre: Skipped 1 previous similar message [ 8821.265938] Lustre: Skipped 4 previous similar messages [ 8835.040072] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8835.212404] Lustre: server umount lustre-OST0000 complete [ 8845.155616] Lustre: server umount lustre-MDT0000 complete [ 8846.816140] LustreError: 244979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8846.831048] LustreError: 244979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8848.541391] LustreError: 246822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789643837 with bad export cookie 13873411435442779026 [ 8848.541861] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8848.554831] LustreError: 246822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8848.788096] Lustre: server umount lustre-MDT0001 complete [ 8943.304763] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [ 8946.385318] Key type lgssc unregistered [ 8946.648705] LNet: 248583:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8946.654964] LNetError: 248583:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8946.670887] LNet: Removed LNI 192.168.203.139@tcp [ 8947.549140] Key type .llcrypt unregistered [ 8947.552935] Key type ._llcrypt unregistered [ 8974.121343] Key type ._llcrypt registered [ 8974.124314] Key type .llcrypt registered [ 8974.231634] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 8989.649550] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 8990.726113] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8990.789679] alg: No test for adler32 (adler32-zlib) [ 8991.809271] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [ 8992.025528] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [ 8993.711218] Key type lgssc registered [ 8994.712947] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9040.903415] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9051.618825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9051.661408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.007738] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9053.058661] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9053.152428] Lustre: lustre-MDT0000: new disk, initializing [ 9053.223204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9053.239725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9058.316516] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9068.769292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9068.923988] Lustre: 252656:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9069.021087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9069.026210] Lustre: Skipped 1 previous similar message [ 9069.133052] Lustre: lustre-MDT0001: new disk, initializing [ 9069.227847] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9069.270965] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9069.284458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9073.411496] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9077.894316] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9084.329848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9084.652605] Lustre: lustre-OST0000: new disk, initializing [ 9084.657974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9084.664798] Lustre: 254288:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9084.781871] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9087.654600] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9087.666367] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9087.797747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9090.937066] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9101.725177] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9101.913113] Lustre: lustre-OST0001: new disk, initializing [ 9101.916649] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9101.924084] Lustre: 255159:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9101.996986] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9108.187411] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9109.617645] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9109.632783] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9109.742746] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9119.040919] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9125.837344] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9140.708947] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9140.719066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9140.722616] Lustre: Skipped 2 previous similar messages [ 9140.732853] Lustre: Skipped 1 previous similar message [ 9142.608720] Lustre: server umount lustre-MDT0000 complete [ 9146.109027] LustreError: 252647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644135 with bad export cookie 8705078852611071311 [ 9146.115973] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9146.124774] LustreError: 252647:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9146.578847] Lustre: server umount lustre-MDT0001 complete [ 9160.312436] Lustre: server umount lustre-OST0000 complete [ 9174.256401] Lustre: server umount lustre-OST0001 complete [ 9188.075949] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9198.456495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9206.116979] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9215.420556] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) [ 9234.549786] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9245.590111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9245.618100] Lustre: lustre-MDT0000: reset Object Index mappings [ 9246.091325] LustreError: 260165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9246.119595] LustreError: 260165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9246.183979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9250.761977] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9251.302724] LustreError: 260166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9256.422733] LustreError: 260165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9259.966461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9259.980586] Lustre: lustre-MDT0001: reset Object Index mappings [ 9260.443388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9265.165370] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9268.043794] Lustre: 261307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9274.536757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9280.639686] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9285.099859] LustreError: 261661:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9288.747223] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9289.048242] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9289.056371] Lustre: Skipped 1 previous similar message [ 9291.134316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 9291.204584] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:67 to 0x2c0000401:97) [ 9296.254818] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9304.514131] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9309.266822] Lustre: 263182:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9316.731692] Lustre: server umount lustre-MDT0000 complete [ 9316.836216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9316.843389] Lustre: Skipped 1 previous similar message [ 9316.851603] LustreError: 260165:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9316.858230] LustreError: 260165:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9320.472347] LustreError: 260147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644309 with bad export cookie 8705078852611078101 [ 9320.474365] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9320.494429] LustreError: 260147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9321.026827] Lustre: server umount lustre-MDT0001 complete [ 9335.001214] Lustre: server umount lustre-OST0000 complete [ 9338.346018] Lustre: 250164:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1789644311/real 1789644311] req@ffff98203b9c9500 x1876577727147136/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1789644327 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 [ 9338.358895] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.364446] Lustre: Skipped 3 previous similar messages [ 9338.601754] Lustre: server umount lustre-OST0001 complete [ 9355.656338] Lustre: DEBUG MARKER: == conf-sanity test 155: gap in seq allocation from ofd after restarting ========================================================== 07:25:44 (1789644344) [ 9371.795469] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9382.690294] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9417.169868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9417.386981] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9417.413339] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9417.476258] Lustre: lustre-MDT0000: new disk, initializing [ 9417.559502] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9417.571489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9421.651790] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9431.765092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9431.865482] Lustre: 267435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9431.911336] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9431.915435] Lustre: Skipped 1 previous similar message [ 9431.973338] Lustre: lustre-MDT0001: new disk, initializing [ 9432.075202] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9432.086627] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9436.339727] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9445.433534] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9450.455516] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9456.046808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9456.344748] Lustre: lustre-OST0000: new disk, initializing [ 9456.351105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9456.360109] Lustre: 269401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9457.628318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9457.651023] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9457.771744] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9462.451968] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9471.543731] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9475.924321] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9476.188065] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9480.725606] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9480.883950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9483.240487] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9483.240933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9483.242261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9483.255540] LustreError: Skipped 1 previous similar message [ 9483.284410] Lustre: Skipped 1 previous similar message [ 9488.357342] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9488.361919] Lustre: Skipped 2 previous similar messages [ 9488.701179] Lustre: server umount lustre-OST0000 complete [ 9493.479714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9493.489154] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9493.491432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9493.507208] Lustre: Skipped 1 previous similar message [ 9493.527959] Lustre: Skipped 1 previous similar message [ 9498.301681] Lustre: server umount lustre-MDT0000 complete [ 9498.592530] LustreError: 267444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9498.606967] LustreError: 267444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9501.262778] LustreError: 267426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644490 with bad export cookie 8705078852611080194 [ 9501.270433] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9501.273834] LustreError: 267426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9501.570674] Lustre: server umount lustre-MDT0001 complete [ 9521.222995] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9530.629275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9530.962409] LustreError: 272334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9531.016510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9531.024108] Lustre: Skipped 2 previous similar messages [ 9534.762802] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9542.957962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9548.051209] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9550.750474] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 9557.018636] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9562.967728] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9570.953366] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9571.096709] Lustre: lustre-OST0001: new disk, initializing [ 9571.100826] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9571.107669] Lustre: 274637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9572.974687] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9572.985722] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9573.121655] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9576.930586] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9586.757332] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9590.229544] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9595.693774] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000401 to 0x280000402 [ 9595.814935] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000401 to 0x2c0000402 [ 9604.577643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9604.584364] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9604.595310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9610.617838] Lustre: server umount lustre-MDT0000 complete [ 9613.287362] LustreError: 272334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9613.309853] LustreError: 272334:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 9613.851837] LustreError: 272317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644602 with bad export cookie 8705078852611081076 [ 9613.863811] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9613.867194] LustreError: 272317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9614.180803] Lustre: server umount lustre-MDT0001 complete [ 9628.772983] Lustre: server umount lustre-OST0000 complete [ 9642.564099] Lustre: server umount lustre-OST0001 complete [ 9660.864651] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9670.959363] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9671.335533] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9671.340595] Lustre: Skipped 3 previous similar messages [ 9676.062569] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9681.901245] LustreError: 277988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9681.925105] LustreError: 277988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9683.423033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9688.466911] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9691.173214] Lustre: 279127:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 [ 9697.129799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9703.519908] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9713.561797] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9716.043237] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:4 to 0x280000402:65) [ 9716.108468] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:65) [ 9721.890476] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9729.509167] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9733.034164] Lustre: 281004:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify general/lod.*.mdt_hash=crush (mode = 0) failed: rc = -17 [ 9738.534667] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000403 [ 9738.598118] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x280000402 to 0x280000403 [ 9751.474168] Lustre: DEBUG MARKER: == conf-sanity test 156: root_fid on export consistent with client mount ========================================================== 07:32:20 (1789644740) [ 9765.857027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9765.872492] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9765.891410] Lustre: Skipped 3 previous similar messages [ 9765.898913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9765.902045] Lustre: Skipped 4 previous similar messages [ 9772.126393] Lustre: server umount lustre-MDT0000 complete [ 9775.615637] LustreError: 277969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644764 with bad export cookie 8705078852611087446 [ 9775.627267] LustreError: 277969:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9775.628305] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9782.198805] Lustre: server umount lustre-MDT0001 complete [ 9792.386354] Lustre: server umount lustre-OST0000 complete [ 9796.686075] Lustre: server umount lustre-OST0001 complete [ 9805.577379] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [ 9815.699461] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9864.190529] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [ 9874.638538] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9874.941579] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9874.988965] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9875.083442] Lustre: lustre-MDT0000: new disk, initializing [ 9875.257759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9881.427552] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9894.980869] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9895.156957] Lustre: 285532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9895.396661] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9900.789129] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9912.873481] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9913.270645] Lustre: 287163:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9915.257137] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9919.087200] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9929.061403] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9929.280767] Lustre: 288035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9929.420410] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9929.428866] Lustre: Skipped 6 previous similar messages [ 9930.916291] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9930.927283] Lustre: Skipped 1 previous similar message [ 9931.033665] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9935.541833] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [ 9944.583657] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 9948.344378] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 9948.348296] Lustre: Skipped 1 previous similar message [ 9964.192870] Lustre: DEBUG MARKER: == conf-sanity test 157a: test allow_register for MDT registration and MGS restart ========================================================== 07:35:52 (1789644952) [ 9967.083576] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9967.086258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9967.095533] Lustre: Skipped 7 previous similar messages [ 9967.120099] Lustre: Skipped 11 previous similar messages [ 9968.764984] Lustre: server umount lustre-MDT0000 complete [ 9972.205741] LustreError: 285542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9972.224182] LustreError: 285542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 12 previous similar messages [ 9972.491169] LustreError: 285522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789644961 with bad export cookie 8705078852611095377 [ 9972.497991] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9972.503662] LustreError: 285522:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9972.883287] Lustre: server umount lustre-MDT0001 complete [ 9986.706754] Lustre: server umount lustre-OST0000 complete [10000.445965] Lustre: server umount lustre-OST0001 complete [10009.457815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10014.235662] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10022.570587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10027.992064] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10036.044850] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10041.937593] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10049.054209] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10055.727732] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10063.102380] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10084.077195] loop1: detected capacity change from 0 to 204800000 [10084.304858] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10084.323951] blk_update_request: operation not supported error, dev loop1, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10084.360419] blk_update_request: operation not supported error, dev loop1, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10089.205412] loop1: detected capacity change from 0 to 204800000 [10095.739170] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10095.871132] LustreError: 290273:0:(mgs_llog.c:674:mgs_find_or_make_fsdb_nolock()) fs12704: New filesystem registration disabled. Use 'lctl set_param allow_register=1' to allow new filesystem registrations: rc = -13 [10095.886353] LustreError: 290273:0:(mgs_llog.c:733:mgs_check_index()) Can't get db for fs12704 [10095.896673] LustreError: 290273:0:(mgs_handler.c:517:mgs_target_reg()) fs12704-MDT0001: Error checking target index: rc = -13 [10095.910689] LustreError: fs12704-MDT0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10095.925147] LustreError: 294002:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10095.942162] LustreError: 294002:0:(tgt_mount.c:2128:server_put_super()) no obd fs12704-MDT0001 [10095.946577] LustreError: 294002:0:(tgt_mount.c:116:server_deregister_mount()) fs12704-MDT0001 not registered [10096.000735] Lustre: server umount fs12704-MDT0001 complete [10096.004381] LustreError: 294002:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10105.013140] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10105.241929] Lustre: Setting parameter fs12704-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs12704-MDT0001 [10105.319221] Lustre: srv-fs12704-MDT0001: No data found on store. Initialize space. [10105.326441] Lustre: Skipped 4 previous similar messages [10105.406444] Lustre: fs12704-MDT0001: new disk, initializing [10105.408828] Lustre: Skipped 3 previous similar messages [10105.504722] Lustre: srv-fs12704-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [10105.520105] Lustre: 294468:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12704-MDT0001-mdtlov: can't create new llog: rc = -115 [10105.537577] LustreError: 294468:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12704-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10109.983195] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10117.431674] Lustre: server umount fs12704-MDT0001 complete [10132.533489] loop1: detected capacity change from 0 to 204800000 [10134.504455] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10140.567306] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10141.496016] Lustre: 296037:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12704-MDT0001-mdtlov: can't create new llog: rc = -115 [10141.503074] LustreError: 296037:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12704-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10146.971708] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10149.878311] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10153.471285] Lustre: server umount fs12704-MDT0001 complete [10165.225851] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10165.248743] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10165.253687] Lustre: Skipped 3 previous similar messages [10165.263045] Lustre: Skipped 2 previous similar messages [10169.683497] Lustre: server umount lustre-MDT0000 complete [10178.492178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10178.619414] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10184.148577] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10184.184632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10192.560098] loop1: detected capacity change from 0 to 204800000 [10199.552264] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10200.102620] Lustre: 298481:0:(lod_sub_object.c:951:lod_sub_prep_llog()) fs12704-MDT0001-mdtlov: can't create new llog: rc = -115 [10200.122710] LustreError: 298481:0:(lod_dev.c:511:lod_sub_recovery_thread()) fs12704-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [10206.781638] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10214.186571] Lustre: server umount fs12704-MDT0001 complete [10230.260065] LustreError: 292114:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10230.274044] LustreError: 292114:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 110 previous similar messages [10240.484043] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10240.489460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10240.505608] Lustre: Skipped 2 previous similar messages [10244.536563] Lustre: server umount lustre-OST0000 complete [10255.137450] Lustre: server umount lustre-MDT0000 complete [10260.631200] LustreError: 291027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645249 with bad export cookie 8705078852611098058 [10260.642346] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10260.658517] LustreError: 291027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10261.622566] Lustre: server umount lustre-MDT0001 complete [10277.908055] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10280.242335] Key type lgssc unregistered [10280.451820] LNet: 300827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10280.458292] LNetError: 300827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10280.469630] LNet: Removed LNI 192.168.203.139@tcp [10281.308191] Key type .llcrypt unregistered [10281.311718] Key type ._llcrypt unregistered [10303.329403] Key type ._llcrypt registered [10303.331133] Key type .llcrypt registered [10303.523430] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10304.408531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10304.568163] alg: No test for adler32 (adler32-zlib) [10305.627204] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [10305.873124] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10307.551693] Key type lgssc registered [10308.444773] Lustre: Echo OBD driver; http://www.lustre.org/ [10318.069175] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10318.121044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10319.876109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10324.811732] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10329.082434] LustreError: 302100:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10329.099813] LustreError: 302100:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10334.177400] LustreError: 302099:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10334.822066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10335.271732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10339.686511] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10348.191411] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10354.898590] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10361.886699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10362.258393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10364.333138] LustreError: 303930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10369.228100] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10369.512467] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10374.632533] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10377.893277] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10385.908943] LustreError: 303930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10385.928657] LustreError: 303930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10401.390757] loop1: detected capacity change from 0 to 204800000 [10401.646336] blk_update_request: operation not supported error, dev loop1, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10401.663229] blk_update_request: operation not supported error, dev loop1, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10401.686459] blk_update_request: operation not supported error, dev loop1, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10406.384575] LustreError: 303930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.401719] LustreError: 303930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [10407.214857] loop1: detected capacity change from 0 to 204800000 [10413.961089] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10414.183343] Lustre: fs8152-OST0000: new disk, initializing [10414.191381] Lustre: srv-fs8152-OST0000: No data found on store. Initialize space. [10414.197728] Lustre: 305834:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10414.293498] Lustre: fs8152-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10420.947283] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10429.184867] Lustre: server umount fs8152-OST0000 complete [10441.860177] loop1: detected capacity change from 0 to 204800000 [10442.216388] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10442.239319] LustreError: 303931:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 20 previous similar messages [10442.248868] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10448.764712] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10449.131141] Lustre: fs8152-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10456.901089] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10462.696902] LustreError: lustre-OST0001-osc-MDT0001: operation ost_connect to node 0@lo failed: rc = -19 [10466.094660] loop2: detected capacity change from 0 to 204800000 [10466.481286] blk_update_request: operation not supported error, dev loop2, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10466.513108] blk_update_request: operation not supported error, dev loop2, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10466.549402] blk_update_request: operation not supported error, dev loop2, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10473.655897] loop2: detected capacity change from 0 to 204800000 [10479.652278] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10479.865462] LustreError: fs8152-OST0001: New target registration disabled. Use 'lctl set_param allow_register=1' to allow new target registrations. [10479.873618] LustreError: fs8152-OST0001: the MGS refuses to allow this server to start: rc = -13. Please see messages on the MGS. [10479.884890] LustreError: 308875:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -13 [10479.890899] LustreError: 308875:0:(tgt_mount.c:2128:server_put_super()) no obd fs8152-OST0001 [10479.898667] LustreError: 308875:0:(tgt_mount.c:116:server_deregister_mount()) fs8152-OST0001 not registered [10479.987469] Lustre: server umount fs8152-OST0001 complete [10479.996552] LustreError: 308875:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -13 [10493.648239] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10493.937675] Lustre: fs8152-OST0001: new disk, initializing [10493.941434] Lustre: srv-fs8152-OST0001: No data found on store. Initialize space. [10493.950521] Lustre: 309427:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10494.059823] Lustre: fs8152-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10502.387841] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10508.773669] LustreError: 303930:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10508.791814] LustreError: 303930:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 38 previous similar messages [10512.629628] Lustre: server umount fs8152-OST0000 complete [10523.729544] Lustre: server umount fs8152-OST0001 complete [10538.611346] loop1: detected capacity change from 0 to 204800000 [10544.732678] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10544.924547] Lustre: fs8152-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10554.635602] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10564.484916] loop2: detected capacity change from 0 to 204800000 [10570.567811] LDISKFS-fs (dm-6): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10577.622407] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10589.374755] Lustre: server umount fs8152-OST0000 complete [10595.822229] LustreError: lustre-OST0001-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -19 [10599.526567] Lustre: server umount fs8152-OST0001 complete [10615.265835] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10615.282213] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10615.298413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10616.292133] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10616.295569] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.710240] Lustre: server umount lustre-OST0000 complete [10626.528759] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10626.554924] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10626.560660] Lustre: Skipped 1 previous similar message [10629.208082] Lustre: server umount lustre-MDT0000 complete [10633.552557] LustreError: 302081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645622 with bad export cookie 4372726682743393366 [10633.563241] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10633.580694] LustreError: 302081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10634.039631] Lustre: server umount lustre-MDT0001 complete [10645.897352] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10648.582973] Key type lgssc unregistered [10648.864949] LNet: 315350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10648.869694] LNetError: 315350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10648.882339] LNet: Removed LNI 192.168.203.139@tcp [10649.390148] Key type .llcrypt unregistered [10649.392083] Key type ._llcrypt unregistered [10686.374676] Key type ._llcrypt registered [10686.376834] Key type .llcrypt registered [10686.483718] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10703.245905] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10704.260779] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10704.298171] alg: No test for adler32 (adler32-zlib) [10705.275836] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [10705.437636] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10707.095178] Key type lgssc registered [10709.017449] Lustre: Echo OBD driver; http://www.lustre.org/ [10752.117806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10752.139213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10753.310971] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10753.340449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10753.414475] Lustre: lustre-MDT0000: new disk, initializing [10753.509865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10753.545060] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10758.210092] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10770.210028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10770.439759] Lustre: 319207:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10770.523437] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10770.529584] Lustre: Skipped 1 previous similar message [10770.623996] Lustre: lustre-MDT0001: new disk, initializing [10770.728595] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10770.788370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10770.803162] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10775.897179] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10787.082015] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10793.655973] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10801.146312] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10801.602470] Lustre: lustre-OST0000: new disk, initializing [10801.610040] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10801.619508] Lustre: 321177:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10801.769386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10807.936603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10807.968261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10808.080705] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [10808.102876] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10817.931921] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10820.854483] LustreError: 317382:0:(lmv_obd.c:211:lmv_notify()) activation of lustre-MDT0000_UUID failed: -22 [10820.920527] Lustre: Mounted lustre-client - version 2.17.58_39_ga8a99f2 [10821.876328] Lustre: Unmounted lustre-client [10833.889029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10833.889669] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10833.895807] LustreError: Skipped 1 previous similar message [10833.900092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10833.947528] Lustre: Skipped 1 previous similar message [10836.454173] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10836.467029] Lustre: Skipped 2 previous similar messages [10841.577387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10841.584563] Lustre: Skipped 1 previous similar message [10846.691348] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10846.712072] Lustre: Skipped 1 previous similar message [10847.199107] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10847.418385] Lustre: server umount lustre-OST0000 complete [10851.814451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10851.822686] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10851.832291] Lustre: Skipped 1 previous similar message [10857.251970] Lustre: server umount lustre-MDT0000 complete [10860.812101] LustreError: 319199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789645849 with bad export cookie 14988306110083546958 [10860.821837] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10860.824727] LustreError: 319199:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10861.098297] Lustre: server umount lustre-MDT0001 complete [10871.249382] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [10874.181267] Key type lgssc unregistered [10874.400243] LNet: 1 local NIs in recovery (showing 1): 192.168.203.139@tcp [10874.457717] LNet: 323336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10874.469294] LNetError: 323336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10950.245808] LNet: Removed LNI 192.168.203.139@tcp [10951.052683] Key type .llcrypt unregistered [10951.060412] Key type ._llcrypt unregistered [10967.703943] Key type ._llcrypt registered [10967.706598] Key type .llcrypt registered [10967.805270] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [10981.647425] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [10982.782364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10982.799670] alg: No test for adler32 (adler32-zlib) [10983.838541] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [10984.071756] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [10985.727455] Key type lgssc registered [10986.856891] Lustre: Echo OBD driver; http://www.lustre.org/ [11019.356600] Lustre: DEBUG MARKER: == conf-sanity test 161: test '-o mgsname' option ======== 07:53:27 (1789646007) [11026.220581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11026.288926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11027.578528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11027.617763] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11027.733887] Lustre: lustre-MDT0000: new disk, initializing [11027.866499] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11027.891198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11032.004560] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11043.663541] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11043.801837] Lustre: 326554:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11043.848590] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11043.854440] Lustre: Skipped 1 previous similar message [11043.919449] Lustre: lustre-MDT0001: new disk, initializing [11044.004971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11044.041145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11044.060041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11048.304145] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11057.039312] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11062.466910] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11069.545250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11069.739892] Lustre: lustre-OST0000: new disk, initializing [11069.743741] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11069.749289] Lustre: 328522:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11069.840893] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11074.960331] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11076.673093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11076.679383] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11076.799083] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11084.077699] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11097.571321] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.596572] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11097.601774] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11097.617720] Lustre: Skipped 1 previous similar message [11102.697276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11102.702088] Lustre: Skipped 1 previous similar message [11103.913917] Lustre: server umount lustre-OST0000 complete [11110.880479] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11110.894856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11110.915896] Lustre: Skipped 1 previous similar message [11110.925123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11112.932219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11112.932688] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11114.531909] Lustre: server umount lustre-MDT0000 complete [11118.197709] LustreError: 326546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646107 with bad export cookie 2931718825778902592 [11118.203670] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11118.211606] LustreError: 326546:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11118.571950] Lustre: server umount lustre-MDT0001 complete [11130.562830] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11133.088803] Key type lgssc unregistered [11133.410495] LNet: 330429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11133.422880] LNetError: 330429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11133.435854] LNet: Removed LNI 192.168.203.139@tcp [11134.395546] Key type .llcrypt unregistered [11134.397352] Key type ._llcrypt unregistered [11157.674356] Key type ._llcrypt registered [11157.677523] Key type .llcrypt registered [11157.828207] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11159.446081] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11159.524648] alg: No test for adler32 (adler32-zlib) [11160.688675] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [11161.050306] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11162.729877] Key type lgssc registered [11164.493969] Lustre: Echo OBD driver; http://www.lustre.org/ [11172.296747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11172.317719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11173.773480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11178.123743] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11183.075384] LustreError: 331700:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11183.127101] LustreError: 331700:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11188.193944] LustreError: 331701:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11188.248238] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11188.667521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11192.879700] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11200.541177] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11206.654640] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11215.109299] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11215.728868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11224.338517] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11231.669280] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11244.600904] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11246.424525] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11253.879729] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.39@tcp (stopping) [11254.754187] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11254.771151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11254.784342] Lustre: Skipped 2 previous similar messages [11256.771684] Lustre: server umount lustre-MDT0000 complete [11258.941036] LustreError: 331695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11260.947707] LustreError: 331680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646250 with bad export cookie 6112021184942233553 [11260.952378] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11260.955942] LustreError: 331680:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11264.058713] LustreError: 334618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.39@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11264.082298] LustreError: 334618:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11264.089776] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.39@tcp (stopping) [11265.004313] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11265.024775] Lustre: Skipped 2 previous similar messages [11267.431649] Lustre: server umount lustre-MDT0001 complete [11271.572879] Lustre: server umount lustre-OST0000 complete [11280.285381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11285.015379] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11287.730426] Lustre: 335883:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.139@tcp [11289.683791] Lustre: server umount MGS complete [11297.660513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11298.054521] LustreError: 336454:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11298.073728] LustreError: 336454:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11298.129382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11302.939869] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11308.517971] LustreError: 336455:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11308.546553] LustreError: 336455:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11311.807827] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11312.092470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11316.746673] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11326.420339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11326.858114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11333.282340] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11340.732794] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11342.198059] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in IDLE state after 0 sec [11347.855131] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11349.285604] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11353.569896] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11353.578969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11353.582439] Lustre: Skipped 1 previous similar message [11353.584056] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11353.601023] Lustre: Skipped 3 previous similar messages [11355.749982] Lustre: server umount lustre-OST0000 complete [11363.296227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11363.311133] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11363.336676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11365.511604] Lustre: server umount lustre-MDT0000 complete [11368.801524] LustreError: 336434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646357 with bad export cookie 6112021184942234400 [11368.810416] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11368.819213] LustreError: 336434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11369.179673] Lustre: server umount lustre-MDT0001 complete [11379.213254] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11381.478508] Key type lgssc unregistered [11381.777425] LNet: 339890:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11381.786503] LNetError: 339890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11381.816351] LNet: Removed LNI 192.168.203.139@tcp [11382.588550] Key type .llcrypt unregistered [11382.590949] Key type ._llcrypt unregistered [11399.550848] Key type ._llcrypt registered [11399.553979] Key type .llcrypt registered [11399.654478] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [11414.091093] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11415.291307] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11415.305451] alg: No test for adler32 (adler32-zlib) [11416.377619] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [11416.612164] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [11418.311212] Key type lgssc registered [11419.379114] Lustre: Echo OBD driver; http://www.lustre.org/ [11454.822091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11454.872371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11456.159970] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11456.206641] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11456.369860] Lustre: lustre-MDT0000: new disk, initializing [11456.494813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11456.517878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11460.964203] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11471.710820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11471.980397] Lustre: 343050:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11472.201397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11472.220025] Lustre: Skipped 1 previous similar message [11472.417638] Lustre: lustre-MDT0001: new disk, initializing [11472.574930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11472.616530] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11472.646279] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11478.008673] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11487.311732] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11492.399987] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11499.090256] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11499.474575] Lustre: lustre-OST0000: new disk, initializing [11499.479334] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11499.486823] Lustre: 345030:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11499.553479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11505.224662] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11508.827127] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11508.841057] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11508.948324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11514.709988] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11520.525245] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11520.907596] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11526.197552] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11526.439614] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11529.698197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11529.712312] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11529.715554] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11529.734210] Lustre: Skipped 1 previous similar message [11534.748361] Lustre: server umount lustre-OST0000 complete [11539.424482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11539.437318] LustreError: Skipped 1 previous similar message [11539.443910] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11539.477127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11539.484428] Lustre: Skipped 1 previous similar message [11544.865966] Lustre: server umount lustre-MDT0000 complete [11545.057792] LustreError: 343060:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.076155] LustreError: 343060:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11549.158680] LustreError: 344121:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11549.181253] LustreError: 344121:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11549.280695] LustreError: 345029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646538 with bad export cookie 960767587151137136 [11549.289202] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11549.293234] LustreError: 345029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11549.767150] Lustre: server umount lustre-MDT0001 complete [11559.436067] Lustre: DEBUG MARKER: == conf-sanity test 164: test expected_clients parameter and max client tracking ========================================================== 08:02:28 (1789646548) [11579.370314] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [11597.679270] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [11632.494449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11632.763695] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11632.793916] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11632.857149] Lustre: lustre-MDT0000: new disk, initializing [11632.913832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11632.936103] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11637.340747] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11647.191117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11647.341738] Lustre: 349812:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11647.395524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11647.399677] Lustre: Skipped 1 previous similar message [11647.510407] Lustre: lustre-MDT0001: new disk, initializing [11647.614749] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11647.647150] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11647.667328] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11652.027693] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11661.898298] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11667.505329] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11675.033482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11675.393862] Lustre: lustre-OST0000: new disk, initializing [11675.397284] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11675.403941] Lustre: 351780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11675.515754] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11682.007815] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [11682.352260] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11682.358967] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11682.514500] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11691.478623] Lustre: DEBUG MARKER: oleg339-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11723.746199] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11723.748829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11723.759205] LustreError: Skipped 1 previous similar message [11723.765823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11723.768716] Lustre: Skipped 2 previous similar messages [11723.793845] Lustre: Skipped 2 previous similar messages [11728.866962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11728.872467] Lustre: Skipped 1 previous similar message [11729.726953] Lustre: server umount lustre-OST0000 complete [11733.985536] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11734.007223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11734.017089] Lustre: Skipped 1 previous similar message [11739.844176] Lustre: server umount lustre-MDT0000 complete [11743.744226] LustreError: 350605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789646732 with bad export cookie 960767587151138018 [11743.751306] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11743.753947] LustreError: 350605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11744.086685] Lustre: server umount lustre-MDT0001 complete [11754.866421] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11757.681579] Key type lgssc unregistered [11757.948746] LNet: 353951:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11757.958953] LNetError: 353951:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11757.971902] LNet: Removed LNI 192.168.203.139@tcp [11758.851712] Key type .llcrypt unregistered [11758.854873] Key type ._llcrypt unregistered [11850.846614] Key type ._llcrypt registered [11850.848022] Key type .llcrypt registered [11851.008369] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11851.279709] Key type .llcrypt unregistered [11851.281590] Key type ._llcrypt unregistered [11924.163248] Key type ._llcrypt registered [11924.166489] Key type .llcrypt registered [11924.267755] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11924.573327] Key type .llcrypt unregistered [11924.575158] Key type ._llcrypt unregistered [11955.464722] Key type ._llcrypt registered [11955.467287] Key type .llcrypt registered [11955.617835] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11955.956387] Key type .llcrypt unregistered [11955.958662] Key type ._llcrypt unregistered [11985.204478] Key type ._llcrypt registered [11985.208459] Key type .llcrypt registered [11985.394982] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [11985.782410] Key type .llcrypt unregistered [11985.787331] Key type ._llcrypt unregistered [12064.921757] Key type ._llcrypt registered [12064.924805] Key type .llcrypt registered [12065.049519] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12065.415310] Key type .llcrypt unregistered [12065.417371] Key type ._llcrypt unregistered [12094.711145] Key type ._llcrypt registered [12094.718433] Key type .llcrypt registered [12094.856193] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12095.136314] Key type .llcrypt unregistered [12095.138725] Key type ._llcrypt unregistered [12123.661461] Key type ._llcrypt registered [12123.668039] Key type .llcrypt registered [12123.813892] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12124.212570] Key type .llcrypt unregistered [12124.215813] Key type ._llcrypt unregistered [12150.596848] Key type ._llcrypt registered [12150.599661] Key type .llcrypt registered [12150.701586] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12150.932294] Key type .llcrypt unregistered [12150.936727] Key type ._llcrypt unregistered [12178.499176] Key type ._llcrypt registered [12178.501785] Key type .llcrypt registered [12178.596108] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12178.894320] Key type .llcrypt unregistered [12178.900266] Key type ._llcrypt unregistered [12208.868491] Key type ._llcrypt registered [12208.873311] Key type .llcrypt registered [12209.002470] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12209.338518] Key type .llcrypt unregistered [12209.341696] Key type ._llcrypt unregistered [12274.213801] Key type ._llcrypt registered [12274.216170] Key type .llcrypt registered [12274.420930] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12274.896475] Key type .llcrypt unregistered [12274.902639] Key type ._llcrypt unregistered [12304.605274] Key type ._llcrypt registered [12304.607872] Key type .llcrypt registered [12304.720481] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12305.076174] Key type .llcrypt unregistered [12305.078921] Key type ._llcrypt unregistered [12337.177743] Key type ._llcrypt registered [12337.179374] Key type .llcrypt registered [12337.289169] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12337.535474] Key type .llcrypt unregistered [12337.540316] Key type ._llcrypt unregistered [12396.480261] Key type ._llcrypt registered [12396.486857] Key type .llcrypt registered [12396.667297] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [12413.541523] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12414.640032] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12414.672219] alg: No test for adler32 (adler32-zlib) [12415.712024] Lustre: Lustre: Build Version: 2.17.58_39_ga8a99f2 [12416.020433] LNet: Added LNI 192.168.203.139@tcp [8/256/0/180] [12417.783462] Key type lgssc registered [12419.022152] Lustre: Echo OBD driver; http://www.lustre.org/ [12458.196591] Lustre: tst250fs-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12458.232374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12459.407687] Lustre: Setting parameter tst250fs-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log tst250fs-MDT0000 [12459.449660] Lustre: ctl-tst250fs-MDT0000: No data found on store. Initialize space. [12459.581675] Lustre: tst250fs-MDT0000: new disk, initializing [12459.779925] Lustre: tst250fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12459.811369] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12463.687294] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12474.583601] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12474.715863] Lustre: 374217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify tst250fs-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12474.784634] Lustre: srv-tst250fs-MDT0001: No data found on store. Initialize space. [12474.791290] Lustre: Skipped 1 previous similar message [12474.873428] Lustre: tst250fs-MDT0001: new disk, initializing [12475.002454] Lustre: tst250fs-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12475.046122] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12475.070682] Lustre: cli-ctl-tst250fs-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12479.837210] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12485.046837] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [12492.601497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12493.001886] Lustre: tst250fs-OST0000: new disk, initializing [12493.011732] Lustre: srv-tst250fs-OST0000: No data found on store. Initialize space. [12493.019798] Lustre: 375844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12493.153245] Lustre: tst250fs-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12493.374803] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12493.382148] Lustre: cli-tst250fs-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12493.436523] Lustre: tst250fs-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [12499.417471] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12513.274974] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12513.460245] Lustre: tst250fs-OST0001: new disk, initializing [12513.470643] Lustre: srv-tst250fs-OST0001: No data found on store. Initialize space. [12513.486753] Lustre: 376720:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12513.611834] Lustre: tst250fs-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12521.420107] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_default_debug -1 all [12522.702741] Lustre: ctl-tst250fs-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12522.714667] Lustre: cli-tst250fs-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12522.941054] Lustre: tst250fs-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12527.991669] Lustre: Setting parameter tst250fs.osc.tst250fs-*.max_rpcs_in_flight=16 in log params [12530.608233] Lustre: Setting parameter general.osc.*.max_dirty_mb=512 in log params [12530.617339] Lustre: Skipped 2 previous similar messages [12551.648800] LustreError: tst250fs-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12551.656318] Lustre: tst250fs-MDT0000-osp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12551.677441] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12553.698472] Lustre: tst250fs-MDT0000-lwp-MDT0001: Connection to tst250fs-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12553.702956] Lustre: tst250fs-MDT0000: Not available for connect from 0@lo (stopping) [12553.721873] Lustre: Skipped 2 previous similar messages [12553.742506] Lustre: Skipped 3 previous similar messages [12555.784012] Lustre: server umount tst250fs-MDT0000 complete [12559.551943] LustreError: 375843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1789647548 with bad export cookie 15809602653857684427 [12559.564092] LustreError: 375843:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12559.566662] LustreError: MGC192.168.203.139@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12559.866044] Lustre: server umount tst250fs-MDT0001 complete [12573.334543] Lustre: server umount tst250fs-OST0000 complete [12587.579642] Lustre: server umount tst250fs-OST0001 complete [12606.097605] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [12614.142626] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12642.684395] Lustre: DEBUG MARKER: == conf-sanity test 802a: simulate readonly device ======= 08:20:31 (1789647631) [12644.462367] Lustre: DEBUG MARKER: SKIP: conf-sanity test_802a ZFS specific test [12659.661515] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing set_hostid [12668.552246] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing load_modules_local [12698.565900] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12381 sec ========= 08:21:27 (1789647687) [12700.303658] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 08:21:29 (1789647689) === [12704.486519] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 08:21:33 (1789647693) === [12742.344724] Lustre: DEBUG MARKER: oleg339-server.virtnet: executing unload_modules_local [12745.026761] Key type lgssc unregistered [12745.278106] LNet: 385847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12745.285671] LNetError: 385847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12745.307870] LNet: Removed LNI 192.168.203.139@tcp [12746.360180] Key type .llcrypt unregistered [12746.366428] Key type ._llcrypt unregistered